[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 456808206 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2640MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002359] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006958] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.010129] LSM: Security Framework initializing [ 0.012046] Yama: becoming mindful. [ 0.013048] SELinux: Initializing. [ 0.014083] *** VALIDATE selinux *** [ 0.022260] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027779] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030113] *** VALIDATE tmpfs *** [ 0.031578] *** VALIDATE proc *** [ 0.032319] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.034269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035136] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037040] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041278] debug: unmapping init [mem 0xffffffffbc459000-0xffffffffbc460fff] [ 0.043890] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044680] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048021] ... value mask: 0000ffffffffffff [ 0.049026] ... max period: 00007fffffffffff [ 0.050011] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.053201] rcu: Hierarchical SRCU implementation. [ 0.055824] smp: Bringing up secondary CPUs ... [ 0.056577] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.061102] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.225719] node 0 deferred pages initialised in 159ms [ 0.227124] devtmpfs: initialized [ 0.228205] x86/mm: Memory block size: 128MB [ 0.230662] gcov: version magic: 0x41383552 [ 0.232145] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.233086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.234236] pinctrl core: initialized pinctrl subsystem [ 0.235184] [ 0.235667] ************************************************************* [ 0.236025] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237013] ** ** [ 0.238012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.239012] ** ** [ 0.240015] ** This means that this kernel is built to expose internal ** [ 0.241015] ** IOMMU data structures, which may compromise security on ** [ 0.242015] ** your system. ** [ 0.243015] ** ** [ 0.244014] ** If you see this message and you are not debugging the ** [ 0.245017] ** kernel, report this immediately to your vendor! ** [ 0.246014] ** ** [ 0.247013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.248011] ************************************************************* [ 0.249805] NET: Registered protocol family 16 [ 0.250435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.251071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.252059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.253446] cpuidle: using governor menu [ 0.255632] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.257695] PCI: Using configuration type 1 for base access [ 0.260139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.269145] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.272084] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.276032] cryptd: max_cpu_qlen set to 1000 [ 0.278303] ACPI: Added _OSI(Module Device) [ 0.279014] ACPI: Added _OSI(Processor Device) [ 0.280016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.281012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.285259] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.287586] ACPI: Interpreter enabled [ 0.288055] ACPI: PM: (supports S0 S3 S4 S5) [ 0.289010] ACPI: Using IOAPIC for interrupt routing [ 0.290086] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.291450] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.300431] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.301045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.302028] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.303075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.305394] acpiphp: Slot [2] registered [ 0.306127] acpiphp: Slot [5] registered [ 0.307132] acpiphp: Slot [6] registered [ 0.308090] acpiphp: Slot [3] registered [ 0.309149] acpiphp: Slot [4] registered [ 0.310096] acpiphp: Slot [7] registered [ 0.311078] acpiphp: Slot [8] registered [ 0.312076] acpiphp: Slot [9] registered [ 0.313095] acpiphp: Slot [10] registered [ 0.314087] acpiphp: Slot [11] registered [ 0.315116] acpiphp: Slot [12] registered [ 0.316093] acpiphp: Slot [13] registered [ 0.317299] acpiphp: Slot [14] registered [ 0.318097] acpiphp: Slot [15] registered [ 0.319124] acpiphp: Slot [16] registered [ 0.320142] acpiphp: Slot [17] registered [ 0.321123] acpiphp: Slot [18] registered [ 0.322084] acpiphp: Slot [19] registered [ 0.323092] acpiphp: Slot [20] registered [ 0.324119] acpiphp: Slot [21] registered [ 0.325134] acpiphp: Slot [22] registered [ 0.326118] acpiphp: Slot [23] registered [ 0.327107] acpiphp: Slot [24] registered [ 0.328122] acpiphp: Slot [25] registered [ 0.329114] acpiphp: Slot [26] registered [ 0.330098] acpiphp: Slot [27] registered [ 0.331102] acpiphp: Slot [28] registered [ 0.332124] acpiphp: Slot [29] registered [ 0.333109] acpiphp: Slot [30] registered [ 0.334152] acpiphp: Slot [31] registered [ 0.335070] PCI host bridge to bus 0000:00 [ 0.336024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.337035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.340055] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.344190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.346011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348219] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.352747] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.355046] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.357019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.358025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359552] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.360737] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.361050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362809] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.364927] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.369016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.371024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374958] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.382017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.391020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.423000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.445066] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.468019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.490014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.500228] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.504483] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.506379] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.508372] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.510199] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.514196] iommu: Default domain type: Passthrough [ 0.516536] SCSI subsystem initialized [ 0.518139] ACPI: bus type USB registered [ 0.519119] usbcore: registered new interface driver usbfs [ 0.521075] usbcore: registered new interface driver hub [ 0.523079] usbcore: registered new device driver usb [ 0.525195] pps_core: LinuxPPS API ver. 1 registered [ 0.527009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.531057] PTP clock support registered [ 0.533055] EDAC MC: Ver: 3.0.0 [ 0.534387] PCI: Using ACPI for IRQ routing [ 0.536847] NetLabel: Initializing [ 0.538015] NetLabel: domain hash size = 128 [ 0.539013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541127] NetLabel: unlabeled traffic allowed by default [ 0.544006] vgaarb: loaded [ 0.545289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.547013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.555321] clocksource: Switched to clocksource kvm-clock [ 0.670096] VFS: Disk quotas dquot_6.6.0 [ 0.671812] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.674649] *** VALIDATE ramfs *** [ 0.676031] *** VALIDATE hugetlbfs *** [ 0.677724] pnp: PnP ACPI init [ 0.680943] pnp: PnP ACPI: found 6 devices [ 0.698974] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.702598] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.704843] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.706938] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.709660] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.712265] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.715212] NET: Registered protocol family 2 [ 0.717820] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.723196] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.728423] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.733614] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.737161] TCP: Hash tables configured (established 65536 bind 65536) [ 0.740260] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.744143] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.747112] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.750243] NET: Registered protocol family 1 [ 0.752963] RPC: Registered named UNIX socket transport module. [ 0.755464] RPC: Registered udp transport module. [ 0.757242] RPC: Registered tcp transport module. [ 0.759295] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.762091] NET: Registered protocol family 44 [ 0.763825] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.766025] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.768206] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.770506] PCI: CLS 0 bytes, default 64 [ 0.772243] Unpacking initramfs... [ 2.213568] debug: unmapping init [mem 0xffffa0117cc64000-0xffffa0117ffcffff] [ 2.218831] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.220754] software IO TLB: mapped [mem 0x00000000a1000000-0x00000000a5000000] (64MB) [ 2.223124] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.753319] Initialise system trusted keyrings [ 2.761607] Key type blacklist registered [ 2.766285] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.780554] zbud: loaded [ 2.785547] *** VALIDATE nfs *** [ 2.786992] *** VALIDATE nfs4 *** [ 2.790187] pstore: using deflate compression [ 2.794109] Platform Keyring initialized [ 2.919190] NET: Registered protocol family 38 [ 2.921073] Key type asymmetric registered [ 2.922561] Asymmetric key parser 'x509' registered [ 2.924667] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.927909] io scheduler mq-deadline registered [ 2.929994] io scheduler kyber registered [ 2.931295] io scheduler bfq registered [ 2.933157] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.935919] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.938943] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.942210] ACPI: Power Button [PWRF] [ 2.952325] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.961728] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.976907] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.008039] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.039198] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.045281] Non-volatile memory driver v1.3 [ 3.047419] Linux agpgart interface v0.103 [ 3.084148] virtio_blk virtio1: [vda] 145904 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.089618] vda: detected capacity change from 0 to 74702848 [ 3.108581] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.111777] vdb: detected capacity change from 0 to 1073741824 [ 3.119961] libphy: Fixed MDIO Bus: probed [ 3.126374] usbcore: registered new interface driver usbserial_generic [ 3.129524] usbserial: USB Serial support registered for generic [ 3.132445] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.137646] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.139816] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.142623] mousedev: PS/2 mouse device common for all mice [ 3.145632] rtc_cmos 00:05: RTC can wake from S4 [ 3.149208] rtc_cmos 00:05: registered as rtc0 [ 3.150347] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.151447] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.159507] intel_pstate: CPU model not supported [ 3.161293] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.166842] hid: raw HID events driver (C) Jiri Kosina [ 3.169761] usbcore: registered new interface driver usbhid [ 3.170558] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.172404] usbhid: USB HID core driver [ 3.177478] drop_monitor: Initializing network drop monitor service [ 3.180590] Initializing XFRM netlink socket [ 3.182645] NET: Registered protocol family 10 [ 3.186649] Segment Routing with IPv6 [ 3.188325] NET: Registered protocol family 17 [ 3.190327] mpls_gso: MPLS GSO support [ 3.196851] RAS: Correctable Errors collector initialized. [ 3.199482] AVX version of gcm_enc/dec engaged. [ 3.201445] AES CTR mode by8 optimization enabled [ 3.289729] sched_clock: Marking stable (3289645083, 0)->(4238014931, -948369848) [ 3.293762] registered taskstats version 1 [ 3.296351] Loading compiled-in X.509 certificates [ 3.298718] zswap: loaded using pool lzo/zbud [ 3.324181] Key type big_key registered [ 3.336686] Key type encrypted registered [ 3.338032] ima: No TPM chip found, activating TPM-bypass! [ 3.340595] ima: Allocated hash algorithm: sha1 [ 3.342485] ima: No architecture policies found [ 3.344474] evm: Initialising EVM extended attributes: [ 3.346559] evm: security.selinux [ 3.348191] evm: security.ima [ 3.349451] evm: security.capability [ 3.350900] evm: HMAC attrs: 0x1 [ 3.354297] rtc_cmos 00:05: setting system clock to 2026-08-31 18:29:00 UTC (1788200940) [ 3.360683] debug: unmapping init [mem 0xffffffffbd403000-0xffffffffbd5fffff] [ 3.364146] debug: unmapping init [mem 0xffffffffbc182000-0xffffffffbc458fff] [ 3.374072] Write protecting the kernel read-only data: 28672k [ 3.378235] debug: unmapping init [mem 0xffffffffba803000-0xffffffffba9fffff] [ 3.382061] debug: unmapping init [mem 0xffffffffbb114000-0xffffffffbb1fffff] [ 3.438733] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.448575] systemd[1]: Detected virtualization kvm. [ 3.451050] systemd[1]: Detected architecture x86-64. [ 3.453364] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.477645] systemd[1]: No hostname configured. [ 3.479753] systemd[1]: Set hostname to . [ 3.481917] random: systemd: uninitialized urandom read (16 bytes read) [ 3.484725] systemd[1]: Initializing machine ID from random generator. [ 3.626094] random: systemd: uninitialized urandom read (16 bytes read) [ 3.629169] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.635532] random: systemd: uninitialized urandom read (16 bytes read) [ 3.638116] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.642871] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.379771] device-mapper: uevent: version 1.0.3 [ 4.382217] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.258285] virtio_net virtio0 ens2: renamed from eth0 [ 5.395228] scsi host0: ata_piix [ 5.434628] scsi host1: ata_piix [ 5.436352] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.439286] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.008928] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.858868] random: crng init done [ 9.860064] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.386193] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.575500] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.833074] SELinux: Disabled at runtime. [ 11.893425] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.902489] systemd[1]: Detected virtualization kvm. [ 11.904542] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.454798] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.458152] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.463493] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.470277] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.473196] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.479604] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.484159] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root F[ 12.570278] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ile System. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ 13.013365] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.385795] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.424464] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.506895] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.529624] EDAC sbridge: Ver: 1.1.2 [ 14.731749] Key type dns_resolver registered [ 15.050514] NFS: Registering the id_resolver key type [ 15.052744] Key type id_resolver registered [ 15.054505] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg306-client login: [ 44.283043] hrtimer: interrupt took 5060031 ns [ 55.381131] libcfs: loading out-of-tree module taints kernel. [ 55.539296] Key type ._llcrypt registered [ 55.544985] Key type .llcrypt registered [ 56.182344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 56.194296] alg: No test for adler32 (adler32-zlib) [ 57.715334] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 58.633505] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 60.383379] Key type lgssc registered [ 62.200156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 179.905958] Lustre: Mounted lustre-client - version 2.17.56_54_gb664824 [ 184.812448] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 196.152421] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing check_logdir /tmp/testlogs/ [ 201.664849] Lustre: DEBUG MARKER: oleg306-client.virtnet: executing yml_node [ 205.795702] Lustre: lustre-OST0000-osc-ffffa011c50f2800: disconnect after 23s idle [ 206.965802] Lustre: DEBUG MARKER: Client: 2.17.56.54 [ 210.019553] Lustre: DEBUG MARKER: MDS: 2.17.56.54 [ 212.286410] Lustre: DEBUG MARKER: OSS: 2.17.56.54 [ 214.579394] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 31 14:32:29 EDT 2026 [ 232.513589] Lustre: Unmounted lustre-client [ 260.862547] Key type lgssc unregistered [ 261.186329] LNet: 8402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.197084] LNetError: 8402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.209912] LNet: Removed LNI 192.168.203.6@tcp [ 261.781305] Key type .llcrypt unregistered [ 261.783947] Key type ._llcrypt unregistered [ 287.710561] Key type ._llcrypt registered [ 287.719239] Key type .llcrypt registered [ 287.962155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 287.975560] alg: No test for adler32 (adler32-zlib) [ 288.874168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 288.930853] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 288.948422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 292.929456] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 293.043194] LNet: 9284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 293.048164] LNetError: 9284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 293.070569] LNet: Removed LNI 192.168.203.6@tcp [ 293.756412] Key type .llcrypt unregistered [ 293.757861] Key type ._llcrypt unregistered [ 299.756703] Key type ._llcrypt registered [ 299.759497] Key type .llcrypt registered [ 301.112780] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:33:56 (1788201236) [ 301.427862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 301.444950] alg: No test for adler32 (adler32-zlib) [ 302.380448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.479848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.507753] LNet: 9705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.533634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.607261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 302.636507] LNet: 9712:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 308.870933] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:34:04 (1788201244) [ 308.942840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.978390] LNet: 10281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.047984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.156749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 315.748655] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:34:11 (1788201251) [ 315.843401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.875040] LNet: 10867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.902799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.047993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 321.898575] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:34:17 (1788201257) [ 322.109652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.134576] LNet: 11458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.167503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.270598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 328.580855] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:34:24 (1788201264) [ 328.994966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.015837] LNet: 12042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.033960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.173505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 329.790984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 338.084569] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:34:33 (1788201273) [ 338.231632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.267284] LNet: 12628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.305137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.411584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.120320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.811711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 346.525638] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:34:42 (1788201282) [ 346.595447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.646188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 346.726310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 352.801644] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:34:48 (1788201288) [ 353.097387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.162337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.221333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 353.251573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.624786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 353.652280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 354.045115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 354.086595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 354.704814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 354.762570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 355.057070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 355.098611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 355.562426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 355.594966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 356.049398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 356.098960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 356.632235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 356.706078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 357.080374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 357.106969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 357.463580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 357.494666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 357.828568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 357.860365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 358.329554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 358.358210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 358.755562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 358.802662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 364.633498] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:35:00 (1788201300) [ 364.819337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.857167] LNet: 14499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.863565] LNet: 14499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 364.897491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.989290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 365.026519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 371.281925] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:35:07 (1788201307) [ 371.416765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.488987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.598506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 371.647673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 377.556453] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:35:13 (1788201313) [ 377.655068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.706332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.806204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 377.841988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 383.559229] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:35:19 (1788201319) [ 383.656591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.711897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.821729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 383.859956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 389.968986] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:35:25 (1788201325) [ 390.041334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.097373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.182549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 390.219950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 396.078147] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:35:32 (1788201332) [ 396.203303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.244946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.312622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 396.351506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 401.960359] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:35:37 (1788201337) [ 402.134631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.173473] LNet: 18017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.179036] LNet: 18017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 402.226886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.343308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 402.849887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 409.081459] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:35:45 (1788201345) [ 409.295655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.377688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.607829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 409.900298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 417.067678] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:35:52 (1788201352) [ 417.205689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.251934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.489300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.819409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 425.143604] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:36:00 (1788201360) [ 425.226287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.279400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.808511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.143512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.670299] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:36:09 (1788201369) [ 433.809845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.883168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.016722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.068817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 441.069332] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:36:16 (1788201376) [ 441.158051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.192991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.254228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.324578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 448.568848] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:36:24 (1788201384) [ 448.632686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.714492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.794775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 455.689725] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:36:31 (1788201391) [ 455.988872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.048366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.128415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 463.048354] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:36:38 (1788201398) [ 463.136505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.203227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.332965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 463.831658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 470.908612] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:36:46 (1788201406) [ 471.076270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.122190] LNet: 23454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.127434] LNet: 23454:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 471.158027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.286778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 471.343852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 478.771291] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:36:54 (1788201414) [ 478.889459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.953806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.039127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 479.077275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 486.335142] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:37:01 (1788201421) [ 486.421339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.507804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.627961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 486.717363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 493.018350] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:37:08 (1788201428) [ 493.185426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.240367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.374307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 493.415267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 500.240850] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:37:15 (1788201435) [ 500.536361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.582588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.666812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 500.766167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 500.794150] LNetError: 25815:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 500.838704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 500.887667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 506.956605] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:37:22 (1788201442) [ 507.060810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.155384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.253479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LgxNhOukKz [ 507.315264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LgxNhOukKz [ 507.357097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.LgxNhOukKz [ 512.932528] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:37:28 (1788201448) [ 513.118819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.194160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.295817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 513.347442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 513.417714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 519.035747] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:37:34 (1788201454) [ 519.129231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.183571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.323636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 519.363971] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 521.003340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 522.406854] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 524.431294] Key type lgssc registered [ 525.840377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 529.576678] LNetError: 27574:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.6@tcp99 has no available nets [ 532.008612] Key type lgssc unregistered [ 532.345394] LNetError: 28687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.360720] LNet: Removed LNI 192.168.203.6@tcp [ 532.813926] Key type .llcrypt unregistered [ 532.816068] Key type ._llcrypt unregistered [ 538.300253] Key type ._llcrypt registered [ 538.302152] Key type .llcrypt registered [ 539.586916] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:37:55 (1788201475) [ 539.896621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 539.912804] alg: No test for adler32 (adler32-zlib) [ 540.844865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.937912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 543.573179] LNet: Message forwarding will be enabled in 2s [ 546.657274] LNet: Message forwarding enabled [ 547.088767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 547.108479] LNet: Message forwarding will stop in 2s to 6s [ 551.648429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 552.198455] LNet: Message forwarding disabled [ 554.218763] LNet: Message forwarding will be enabled in 2s [ 557.343143] LNet: Message forwarding enabled [ 560.351430] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:38:16 (1788201496) [ 560.414950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.458688] LNet: 30087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.941584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.015706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 568.173495] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:38:24 (1788201504) [ 568.299707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.327134] LNet: 30673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.348420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.409570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 568.454130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 568.486682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 568.511826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 568.538247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 568.569808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 568.593044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 568.624156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 568.662852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 568.696097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 568.726766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 568.754756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 568.782500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 568.812490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 568.845341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 568.881844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 568.923779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 568.960070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 568.996857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 569.010634] LNetError: 30718:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 574.450701] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:38:30 (1788201510) [ 574.609344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.631893] LNet: 31289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.646647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.725388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 581.150702] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:38:36 (1788201516) [ 583.221648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg306-server [ 584.992433] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:38:40 (1788201520) [ 585.105512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.138281] LNet: 32213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.170559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.267735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.293460] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 586.169896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[7-11]@tcp [ 586.206876] LNet: 32352:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 586.298622] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 586.312036] LNetError: 32222:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.8@tcp has gone from up to down [ 587.301063] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.9@tcp is being used as a gateway but routing feature is not turned on [ 587.314384] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 588.332316] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.11@tcp is being used as a gateway but routing feature is not turned on [ 588.344545] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 590.377838] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.9@tcp is being used as a gateway but routing feature is not turned on [ 590.390415] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 592.420898] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:38:48 (1788201528) [ 594.175551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg306-server [ 594.473213] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.8@tcp is being used as a gateway but routing feature is not turned on [ 594.491978] LNetError: 32222:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 595.963792] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:38:51 (1788201531) [ 596.022659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.048685] LNet: 33280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.054507] LNetError: 33280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.075185] LNet: Removed LNI 192.168.203.6@tcp [ 596.114100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.213586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 596.235602] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 596.936433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[7-12/2]@tcp [ 596.974634] LNet: 33405:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 596.982172] LNet: 33405:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 597.014968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[7-12/2]@tcp [ 597.025533] LNetError: 33291:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.9@tcp has gone from up to down [ 597.032553] LNetError: 33291:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 604.346775] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:38:59 (1788201539) [ 604.460293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.492870] LNet: 33993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.499681] LNetError: 33993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.549301] LNet: Removed LNI 192.168.203.6@tcp [ 604.631608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.745973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 604.819260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.878646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.987386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.073092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.131798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.181833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 605.323892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.387308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.500131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.633768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.674537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.733747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 605.794874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.821591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.878092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 605.986641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.038506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.108311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 606.187902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.227781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.306869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.432719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.475403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.539160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 606.683838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.728647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 606.821908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 606.919530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.978576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.064622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 607.138927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.183667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.280629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 614.924932] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:39:10 (1788201550) [ 617.153201] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg306-server [ 619.184502] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:39:14 (1788201554) [ 621.783405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg306-server [ 624.073552] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:39:19 (1788201559) [ 624.305262] LNet: 35397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 624.315390] LNet: 35397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 625.126907] Key type .llcrypt unregistered [ 625.130542] Key type ._llcrypt unregistered [ 631.373558] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 631.437160] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 631.443426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 631.456819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 634.164314] Key type ._llcrypt registered [ 634.166809] Key type .llcrypt registered [ 634.407089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.429144] alg: No test for adler32 (adler32-zlib) [ 635.308674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.364281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.379719] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 635.937978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 635.961294] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 638.023576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 638.060720] LNet: Removed LNI 10.1.2.3@tcp [ 638.229330] LNet: 36126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.238483] LNetError: 36126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.255338] LNet: Removed LNI 192.168.203.6@tcp [ 639.088763] Key type .llcrypt unregistered [ 639.094406] Key type ._llcrypt unregistered [ 651.509826] Key type ._llcrypt registered [ 651.512579] Key type .llcrypt registered [ 652.926454] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:39:48 (1788201588) [ 653.660503] Key type .llcrypt unregistered [ 653.662060] Key type ._llcrypt unregistered [ 660.322467] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 660.461916] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 660.466675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 660.478344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 663.300714] Key type ._llcrypt registered [ 663.302371] Key type .llcrypt registered [ 663.504302] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.515455] alg: No test for adler32 (adler32-zlib) [ 664.537303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.604777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.646226] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 665.560838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 665.605965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 668.170817] LNet: 37629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.176451] LNetError: 37629:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 668.181117] LNet: Removed LNI 10.1.2.3@tcp [ 668.192980] LNet: Removed LNI 192.168.203.6@tcp [ 668.833981] Key type .llcrypt unregistered [ 668.836479] Key type ._llcrypt unregistered [ 680.284550] Key type ._llcrypt registered [ 680.291442] Key type .llcrypt registered [ 681.518701] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:40:17 (1788201617) [ 682.046410] Key type .llcrypt unregistered [ 682.051300] Key type ._llcrypt unregistered [ 687.768453] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.969960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.973788] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.979210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 690.530629] Key type ._llcrypt registered [ 690.532734] Key type .llcrypt registered [ 690.702726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.724138] alg: No test for adler32 (adler32-zlib) [ 691.617419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.672569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 691.695522] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 693.367216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 693.387790] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.067629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 695.087851] LNet: Removed LNI 10.1.2.3@tcp [ 695.103191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 695.140720] LNet: Removed LNI 10.1.2.31@tcp [ 695.164334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 695.186144] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.906672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 695.925172] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 696.859931] LNet: 39557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.871524] LNetError: 39557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 696.877557] LNet: Removed LNI 10.1.2.31@tcp [ 696.896212] LNet: Removed LNI 10.1.2.3@tcp [ 697.490468] Key type .llcrypt unregistered [ 697.493865] Key type ._llcrypt unregistered [ 708.712732] Key type ._llcrypt registered [ 708.715406] Key type .llcrypt registered [ 710.184595] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:40:45 (1788201645) [ 710.670507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 710.691690] alg: No test for adler32 (adler32-zlib) [ 711.647669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.699219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.733140] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 712.487623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.507510] LNet: 40515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.513654] LNetError: 40515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.526896] LNet: Removed LNI 192.168.203.6@tcp [ 712.548931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.606736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.6@tcp [ 712.629925] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 713.582389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.6@tcp [ 713.624966] LNet: Removed LNI 192.168.203.6@tcp [ 713.721522] LNet: 40651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.543541] Key type .llcrypt unregistered [ 714.548873] Key type ._llcrypt unregistered [ 725.996451] Key type ._llcrypt registered [ 726.002787] Key type .llcrypt registered [ 727.620733] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:41:02 (1788201662) [ 729.936324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg306-server [ 731.867760] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:41:07 (1788201667) [ 732.477912] Key type .llcrypt unregistered [ 732.480160] Key type ._llcrypt unregistered [ 739.248904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.385754] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.391733] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.404768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.093814] Key type ._llcrypt registered [ 742.096834] Key type .llcrypt registered [ 742.294836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 742.321970] alg: No test for adler32 (adler32-zlib) [ 743.267258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.328536] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 743.860534] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 745.806708] LNet: 42491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.815342] LNetError: 42491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.819463] LNet: Removed LNI 192.168.203.6@tcp1 [ 745.846132] LNet: Removed LNI 10.1.2.3@tcp2 [ 746.580359] Key type .llcrypt unregistered [ 746.581871] Key type ._llcrypt unregistered [ 757.292849] Key type ._llcrypt registered [ 757.301430] Key type .llcrypt registered [ 759.243702] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:41:34 (1788201694) [ 759.933846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 759.985841] alg: No test for adler32 (adler32-zlib) [ 761.034203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.125990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 761.173410] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 762.592930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.633310] LNet: 43466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.638635] LNetError: 43466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.656771] LNet: Removed LNI 192.168.203.6@tcp [ 762.701533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.761564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 762.816367] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 770.238380] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:41:46 (1788201706) [ 770.571567] LNet: 44058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.577839] LNetError: 44058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.598371] LNet: Removed LNI 192.168.203.6@tcp [ 771.229587] Key type .llcrypt unregistered [ 771.231441] Key type ._llcrypt unregistered [ 777.378423] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.451506] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.460621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.467707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 780.294589] Key type ._llcrypt registered [ 780.298297] Key type .llcrypt registered [ 780.661258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.687903] alg: No test for adler32 (adler32-zlib) [ 781.581722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.661343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.690685] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 782.421935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 782.454624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.706464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.745374] LNet: 44802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.758347] LNetError: 44802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 784.764916] LNet: Removed LNI 10.1.2.3@tcp [ 784.773696] LNet: Removed LNI 192.168.203.6@tcp [ 784.786093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.890986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 784.936559] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 784.951549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.661165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.700256] LNet: 44829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.721792] LNetError: 44829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 785.736778] LNetError: 44829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 785.751768] LNet: Removed LNI 10.1.2.3@tcp [ 785.778635] LNet: Removed LNI 192.168.203.6@tcp [ 785.818307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.902230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 785.969527] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 785.975552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.567209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 786.600339] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 787.762673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.796145] LNet: 44991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.803466] LNetError: 44991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 787.817206] LNetError: 44991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 787.827720] LNet: Removed LNI 10.1.2.3@tcp [ 787.832667] LNet: Removed LNI 192.168.203.6@tcp [ 787.874611] LNet: Removed LNI 192.168.203.6@tcp2 [ 787.913538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.027757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 788.065379] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 788.070817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.076700] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 795.185832] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:42:10 (1788201730) [ 795.332392] LNet: 45588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.349418] LNetError: 45588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.361424] LNetError: 45588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 795.369600] LNet: Removed LNI 10.1.2.3@tcp [ 795.371355] LNet: Removed LNI 192.168.203.6@tcp [ 795.386628] LNet: Removed LNI 192.168.203.6@tcp2 [ 796.075679] Key type .llcrypt unregistered [ 796.083687] Key type ._llcrypt unregistered [ 802.215569] Key type ._llcrypt registered [ 802.218597] Key type .llcrypt registered [ 802.396421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.410377] alg: No test for adler32 (adler32-zlib) [ 803.341963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.400560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.415273] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 804.150968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.172565] LNet: 45988:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.183204] LNetError: 45988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.195856] LNet: Removed LNI 192.168.203.6@tcp [ 804.225449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.303733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 804.330455] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 804.780955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.806948] LNet: 46018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.813784] LNetError: 46018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.823575] LNet: Removed LNI 192.168.203.6@tcp [ 804.851903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.898293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 804.918112] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 805.345425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.366874] LNet: Removed LNI 192.168.203.6@tcp [ 805.385940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.441906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 805.458334] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 805.942989] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 806.064509] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 806.068152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 806.072511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.427596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.464259] LNet: 46127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.476860] LNet: 46127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 808.496505] LNetError: 46127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.507573] LNetError: 46127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 808.537203] LNet: Removed LNI 192.168.203.6@tcp [ 808.578737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.659467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 808.689502] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 815.826207] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:42:31 (1788201751) [ 815.878163] LNet: 46717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.890710] LNetError: 46717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.919698] LNet: Removed LNI 192.168.203.6@tcp [ 816.499884] Key type .llcrypt unregistered [ 816.505287] Key type ._llcrypt unregistered [ 822.334988] Key type ._llcrypt registered [ 822.337086] Key type .llcrypt registered [ 822.492312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.505416] alg: No test for adler32 (adler32-zlib) [ 823.404556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.474788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.511304] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 824.394228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.422324] LNet: 47128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.432567] LNetError: 47128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.453613] LNet: Removed LNI 192.168.203.6@tcp [ 824.472681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.554277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 824.614530] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 831.856435] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:42:47 (1788201767) [ 832.169579] LNet: 47720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.199252] LNetError: 47720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.220757] LNet: Removed LNI 192.168.203.6@tcp [ 833.025807] Key type .llcrypt unregistered [ 833.028515] Key type ._llcrypt unregistered [ 839.133524] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.263932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.270050] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.273463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.000891] Key type ._llcrypt registered [ 842.003876] Key type .llcrypt registered [ 842.239191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.252179] alg: No test for adler32 (adler32-zlib) [ 843.181715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.255432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.293240] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 844.068714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.095591] LNet: 48171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.101583] LNetError: 48171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.114040] LNet: Removed LNI 192.168.203.6@tcp [ 844.127817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.200554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 844.247608] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 851.215449] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:43:06 (1788201786) [ 851.311635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.341427] LNet: 48762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.354516] LNetError: 48762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.376478] LNet: Removed LNI 192.168.203.6@tcp [ 851.398712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.560290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 857.773521] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:43:13 (1788201793) [ 857.924059] LNet: 49348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.551844] Key type .llcrypt unregistered [ 858.555740] Key type ._llcrypt unregistered [ 864.738231] Key type ._llcrypt registered [ 864.740693] Key type .llcrypt registered [ 864.986700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 865.015222] alg: No test for adler32 (adler32-zlib) [ 865.963280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.018865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.048342] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 866.888397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.905201] LNet: 49754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.912393] LNetError: 49754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.929147] LNet: Removed LNI 192.168.203.6@tcp [ 866.945636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.027232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 867.059059] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 873.214867] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:43:29 (1788201809) [ 873.439748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.503134] LNet: 50346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.521671] LNetError: 50346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.553643] LNet: Removed LNI 192.168.203.6@tcp [ 873.575465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.642680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 880.207504] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:43:35 (1788201815) [ 881.884529] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 884.166815] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:43:39 (1788201819) [ 885.907774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 887.999566] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:43:43 (1788201823) [ 889.380555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 891.019042] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:43:46 (1788201826) [ 892.609640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 894.273843] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:43:50 (1788201830) [ 896.074889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 897.846679] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:43:53 (1788201833) [ 898.315811] LNet: 52400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.199795] Key type .llcrypt unregistered [ 899.202906] Key type ._llcrypt unregistered [ 905.315931] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 905.454555] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 905.461420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 905.469963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 908.127235] Key type ._llcrypt registered [ 908.131758] Key type .llcrypt registered [ 908.333426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 908.355891] alg: No test for adler32 (adler32-zlib) [ 909.324760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.448422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 909.481244] LNet: Added LNI 192.168.203.6@tcp [8/512/0/180] [ 909.491669] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 909.495872] LNet: 52721:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 909.630276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.660413] LNet: 52740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.669472] LNetError: 52740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.680526] LNet: Removed LNI 10.1.2.3@tcp [ 909.684958] LNetError: 52740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.697683] LNet: Removed LNI 192.168.203.6@tcp [ 909.732922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.819748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 909.850230] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 909.941726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.972509] LNet: 52764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.978445] LNetError: 52764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.000400] LNet: Removed LNI 192.168.203.6@tcp [ 910.052495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.125146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 910.149475] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 910.155709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.157323] LNet: 52772:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 910.280521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.310801] LNet: 52790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.319395] LNetError: 52790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.330748] LNet: Removed LNI 10.1.2.3@tcp [ 910.334990] LNetError: 52790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.349611] LNet: Removed LNI 192.168.203.6@tcp [ 910.408647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.502337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 910.523645] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 910.527898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 910.530878] LNet: 52798:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 910.758978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.782301] LNet: 52824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.790464] LNetError: 52824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.796815] LNet: Removed LNI 10.1.2.3@tcp [ 910.799635] LNetError: 52824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.817377] LNet: Removed LNI 192.168.203.6@tcp [ 910.860731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.949686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 910.976613] LNet: Added LNI 192.168.203.6@tcp [8/256/0/182] [ 910.989335] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 911.187458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.232863] LNet: 52852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.247840] LNetError: 52852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.261818] LNet: Removed LNI 10.1.2.3@tcp [ 911.265817] LNetError: 52852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.286674] LNet: Removed LNI 192.168.203.6@tcp [ 911.322863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.462803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.509682] LNet: Added LNI 192.168.203.6@tcp [8/256/0/182] [ 911.529041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 911.534244] LNet: 52860:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 911.650987] LNet: Removed LNI 10.1.2.3@tcp [ 911.672207] LNet: Removed LNI 192.168.203.6@tcp [ 912.542072] Key type .llcrypt unregistered [ 912.551981] Key type ._llcrypt unregistered [ 925.449766] Key type ._llcrypt registered [ 925.452443] Key type .llcrypt registered [ 926.762827] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 14:44:22 (1788201862) [ 928.414797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 930.651065] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:44:26 (1788201866) [ 931.092395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 931.114630] alg: No test for adler32 (adler32-zlib) [ 932.140526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.217985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.274437] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 933.353470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.6 [ 933.373310] LNet: 54128:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 933.382797] LNet: Removed LNI 192.168.203.6@tcp [ 940.088440] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:44:35 (1788201875) [ 940.486751] LNet: 54698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.493628] LNetError: 54698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.515136] LNet: Removed LNI 192.168.203.6@tcp [ 941.258651] Key type .llcrypt unregistered [ 941.261438] Key type ._llcrypt unregistered [ 947.811748] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.080472] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 950.788567] Key type ._llcrypt registered [ 950.790950] Key type .llcrypt registered [ 950.936210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.967420] alg: No test for adler32 (adler32-zlib) [ 952.370424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 959.020834] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:44:54 (1788201894) [ 959.168356] LNet: 55613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.174928] LNetError: 55613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 959.208081] LNet: Removed LNI 10.1.2.3@tcp [ 959.866135] Key type .llcrypt unregistered [ 959.868059] Key type ._llcrypt unregistered [ 965.159502] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 965.462876] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 965.466688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 965.469915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 967.270706] Key type ._llcrypt registered [ 967.275517] Key type .llcrypt registered [ 967.460863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 967.484453] alg: No test for adler32 (adler32-zlib) [ 968.751679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 976.902845] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:45:12 (1788201912) [ 978.911824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 981.904418] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:45:16 (1788201916) [ 982.092960] LNet: 56787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.103447] LNetError: 56787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.130631] LNet: Removed LNI 10.1.2.3@tcp [ 983.305800] Key type .llcrypt unregistered [ 983.308496] Key type ._llcrypt unregistered [ 990.720838] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 990.870065] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 990.874036] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 990.877738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.587508] Key type ._llcrypt registered [ 993.590859] Key type .llcrypt registered [ 993.867819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.906419] alg: No test for adler32 (adler32-zlib) [ 994.980114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 995.057179] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1004.636693] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:45:39 (1788201939) [ 1005.315201] LNet: 58030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.330396] LNetError: 58030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1005.352766] LNet: Removed LNI 10.1.2.3@tcp [ 1006.456396] Key type .llcrypt unregistered [ 1006.463836] Key type ._llcrypt unregistered [ 1012.829996] Key type ._llcrypt registered [ 1012.839263] Key type .llcrypt registered [ 1013.069613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1013.086506] alg: No test for adler32 (adler32-zlib) [ 1014.048908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1014.091213] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1022.512595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1023.225266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_interrupt -r 1 [ 1023.256874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_interrupt -r 1 [ 1023.299016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1023.329059] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1023.592666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.629879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.662899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.700463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.748691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.778560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.829590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.862406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.890378] LNet: 58472:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.909691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.957820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.985084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.009909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.064887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.112617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.170815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1024.688489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_dropped -r 1 [ 1024.719644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_dropped -r 1 [ 1024.767975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1024.790227] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1025.006799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.041478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.071158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.108583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.145062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.181889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.216084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.252810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.269530] LNet: 58582:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.292650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.329567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.360984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.376386] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1025.402171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.448935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.491727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.543374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.237731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_aborted -r 1 [ 1026.301389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_aborted -r 1 [ 1026.353955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1026.372810] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1026.382453] LNet: Skipped 1 previous similar message [ 1026.685633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.729313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.758633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.788412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.825373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.852857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.888494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.934614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.958462] LNet: 58691:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1026.986950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.031462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.060901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.084959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.136148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.178712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.203518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1027.697650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_no_route -r 1 [ 1027.763343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_no_route -r 1 [ 1027.840954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1028.120542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.161671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1028.189842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.228325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.256820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.287000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.321880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.360563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.391988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.447681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.486757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.537132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.608772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.640133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.674438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1029.467303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_timeout -r 1 [ 1029.541839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_timeout -r 1 [ 1029.598483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1029.650150] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1029.653488] LNet: Skipped 1 previous similar message [ 1030.011413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.058850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1030.081583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.111633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.136683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.171608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.219409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.238983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.252914] LNet: 58910:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.257942] LNet: 58910:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1030.271984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.309047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.341128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.395076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.449584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.508539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.551558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1031.165122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_error -r 1 [ 1031.224573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_error -r 1 [ 1031.272383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1031.509909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.538652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1031.574901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.615533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.651801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.680700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.728755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.765955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.795401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.827656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.863473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.903314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.937658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.971890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.013692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1032.279660] LNet: 59064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.287665] LNetError: 59064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.331589] LNet: Removed LNI 192.168.203.6@tcp [ 1033.684677] Key type .llcrypt unregistered [ 1033.686721] Key type ._llcrypt unregistered [ 1052.136780] Key type ._llcrypt registered [ 1052.138411] Key type .llcrypt registered [ 1053.242348] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:46:29 (1788201989) [ 1054.387776] Key type .llcrypt unregistered [ 1054.394259] Key type ._llcrypt unregistered [ 1061.010601] Key type ._llcrypt registered [ 1061.015795] Key type .llcrypt registered [ 1061.269390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.306288] alg: No test for adler32 (adler32-zlib) [ 1062.178396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1062.202451] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1070.659572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1081.609677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.661339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1081.699343] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1083.239960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_interrupt -r 1 [ 1083.300577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_interrupt -r 1 [ 1083.363207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_interrupt -r 1 [ 1083.414424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_interrupt -r 1 [ 1083.448333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_interrupt -r 1 [ 1083.479067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_interrupt -r 1 [ 1083.515462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_interrupt -r 1 [ 1083.555231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_interrupt -r 1 [ 1083.596871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1083.627148] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1083.631151] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1083.793413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1083.821527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1083.870514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1083.912134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.967685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.018101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.073894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.126071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.141741] LNet: 60539:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.159360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.190804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.227278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.254633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.310563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.354710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.400938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.938621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_dropped -r 1 [ 1084.968519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_dropped -r 1 [ 1084.997262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_dropped -r 1 [ 1085.038315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_dropped -r 1 [ 1085.081762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_dropped -r 1 [ 1085.135284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_dropped -r 1 [ 1085.165646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_dropped -r 1 [ 1085.209791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_dropped -r 1 [ 1085.247767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1085.265438] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1085.274774] LNet: Skipped 2 previous similar messages [ 1085.488457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.536363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.577228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.620082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.673562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.710654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.751348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.808712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1085.831276] LNet: 60660:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1085.857211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1085.899410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1085.940962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1085.989758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.049729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.097441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.164406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1086.716283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_aborted -r 1 [ 1086.765392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_aborted -r 1 [ 1086.806474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_aborted -r 1 [ 1086.850583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_aborted -r 1 [ 1086.887500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_aborted -r 1 [ 1086.921749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_aborted -r 1 [ 1086.967767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_aborted -r 1 [ 1087.028816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_aborted -r 1 [ 1087.086026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1087.125886] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1087.135228] LNet: Skipped 2 previous similar messages [ 1087.402644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.435275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.463403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.510560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.575816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.628836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.669086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.698485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.737046] LNet: 60782:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.767614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.801543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.829597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.858624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.889203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.912389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.973750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.585282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_no_route -r 1 [ 1088.625613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_no_route -r 1 [ 1088.692633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_no_route -r 1 [ 1088.739633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_no_route -r 1 [ 1088.773029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_no_route -r 1 [ 1088.819660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_no_route -r 1 [ 1088.865171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_no_route -r 1 [ 1088.911559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_no_route -r 1 [ 1088.958336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1089.182424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.213662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.241368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.274494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.296494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.321572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.353723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.396877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.442076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.476237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.510066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.555466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.584856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.620547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.643316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.082169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_timeout -r 1 [ 1090.133238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_timeout -r 1 [ 1090.210900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_timeout -r 1 [ 1090.283664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_timeout -r 1 [ 1090.331294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_timeout -r 1 [ 1090.365098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_timeout -r 1 [ 1090.403950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_timeout -r 1 [ 1090.440460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_timeout -r 1 [ 1090.479582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1090.491768] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1090.496269] LNet: Skipped 5 previous similar messages [ 1090.676489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.710855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.747220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.775782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.819556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.846018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.884081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.910418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.928827] LNet: 61024:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1090.937827] LNet: 61024:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1090.953226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.004090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.041886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.080445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.119437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.161274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.198242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1091.589582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_error -r 1 [ 1091.630700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_error -r 1 [ 1091.668394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_error -r 1 [ 1091.718100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_error -r 1 [ 1091.754278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_error -r 1 [ 1091.810347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_error -r 1 [ 1091.853420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_error -r 1 [ 1091.888256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_error -r 1 [ 1091.940387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1092.168885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.204833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1092.257779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.308677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.331990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.370666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.394169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.418955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.442383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.475587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.500438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.537325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.569639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.603784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.631163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1096.971965] LNet: 61226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.977672] LNetError: 61226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.986174] LNet: Removed LNI 192.168.203.6@tcp [ 1098.021703] LNet: Removed LNI 192.168.203.6@tcp1 [ 1098.484782] Key type .llcrypt unregistered [ 1098.486663] Key type ._llcrypt unregistered [ 1115.651912] Key type ._llcrypt registered [ 1115.656718] Key type .llcrypt registered [ 1117.193438] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:47:32 (1788202052) [ 1118.059524] Key type .llcrypt unregistered [ 1118.061867] Key type ._llcrypt unregistered [ 1124.239250] Key type ._llcrypt registered [ 1124.241431] Key type .llcrypt registered [ 1124.494487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1124.512408] alg: No test for adler32 (adler32-zlib) [ 1125.459869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1125.496106] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1133.458753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1134.011495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_dropped -r 1 [ 1134.037916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_dropped -r 1 [ 1134.060704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1134.096131] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1134.313721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1134.344548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1134.376279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1134.407931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1134.436427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1134.491126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1134.562904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1134.600825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1134.623297] LNet: 62480:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1134.641303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1134.668153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1134.700730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1134.730591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1134.762874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1134.801879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.847308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1135.378727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_error -r 1 [ 1135.406487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_error -r 1 [ 1135.456134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1135.489663] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1135.701331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.729976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.757698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.793708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.837780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.866929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.907156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.954159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.984769] LNet: 62589:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1136.021735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1136.058230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1136.137885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1136.196435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1136.245393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1136.318729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1136.354184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1136.859323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_timeout -r 1 [ 1136.919039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_timeout -r 1 [ 1136.973402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1137.003489] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1137.241272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1137.290533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1137.332795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1137.358534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1137.387846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1137.415569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1137.459201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1137.498860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1137.520609] LNet: 62698:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1137.555390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1137.594671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1137.623922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.648995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1137.697221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1137.738185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1137.762338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.083383] LNet: 62743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.089614] LNetError: 62743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.104873] LNet: Removed LNI 192.168.203.6@tcp [ 1138.479443] Key type .llcrypt unregistered [ 1138.482193] Key type ._llcrypt unregistered [ 1154.482532] Key type ._llcrypt registered [ 1154.485669] Key type .llcrypt registered [ 1155.809110] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:48:11 (1788202091) [ 1156.670556] Key type .llcrypt unregistered [ 1156.673612] Key type ._llcrypt unregistered [ 1162.775591] Key type ._llcrypt registered [ 1162.778145] Key type .llcrypt registered [ 1162.948312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.962335] alg: No test for adler32 (adler32-zlib) [ 1163.915720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1163.952021] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1171.480945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1181.510889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.541478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1181.567478] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1182.999672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_dropped -r 1 [ 1183.030869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e remote_dropped -r 1 [ 1183.061474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_dropped -r 1 [ 1183.087406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e remote_dropped -r 1 [ 1183.112505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e remote_dropped -r 1 [ 1183.148914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e remote_dropped -r 1 [ 1183.181618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e remote_dropped -r 1 [ 1183.220275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e remote_dropped -r 1 [ 1183.252349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1183.278520] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1183.291178] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.106@tcp [ 1183.500440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.524729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.546213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.569339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.600930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.625837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.649964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.675719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.695402] LNet: 64215:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.710386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.737569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.764152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.794474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.831536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.862757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.900974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.936508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.387920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_error -r 1 [ 1184.440772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e remote_error -r 1 [ 1184.480353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_error -r 1 [ 1184.512470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e remote_error -r 1 [ 1184.539825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e remote_error -r 1 [ 1184.570382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e remote_error -r 1 [ 1184.603949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e remote_error -r 1 [ 1184.646820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e remote_error -r 1 [ 1184.693649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1184.725250] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1184.733166] LNet: Skipped 2 previous similar messages [ 1184.924981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.956747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.987857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.018569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.067604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.133879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.185726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.220405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.245570] LNet: 64338:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1185.271826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.310894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.344776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.368390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.397816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.432585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.468795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.511393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.034871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_timeout -r 1 [ 1186.090725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e remote_timeout -r 1 [ 1186.141868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_timeout -r 1 [ 1186.184762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e remote_timeout -r 1 [ 1186.228725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e remote_timeout -r 1 [ 1186.265888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e remote_timeout -r 1 [ 1186.292646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e remote_timeout -r 1 [ 1186.324902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e remote_timeout -r 1 [ 1186.362061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.106@tcp [ 1186.398112] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1186.404462] LNet: Skipped 1 previous similar message [ 1186.627254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.669689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.701403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.736985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.768181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.803770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.834441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.875548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.897718] LNet: 64461:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.918456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.947439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.975339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.006532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.041571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.066807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.093974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1187.128817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.675075] LNet: 64543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.682310] LNetError: 64543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.690124] LNet: Removed LNI 192.168.203.6@tcp [ 1191.707314] LNet: Removed LNI 192.168.203.6@tcp1 [ 1192.235718] Key type .llcrypt unregistered [ 1192.240676] Key type ._llcrypt unregistered [ 1208.206046] Key type ._llcrypt registered [ 1208.212980] Key type .llcrypt registered [ 1209.285892] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:49:05 (1788202145) [ 1210.882980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1212.749369] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:49:08 (1788202148) [ 1213.666825] Key type .llcrypt unregistered [ 1213.670946] Key type ._llcrypt unregistered [ 1220.236311] Key type ._llcrypt registered [ 1220.239861] Key type .llcrypt registered [ 1220.541193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1220.567548] alg: No test for adler32 (adler32-zlib) [ 1221.587901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1221.648743] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1229.526401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1230.273779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e network_timeout -r 1 [ 1230.340152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e network_timeout -r 1 [ 1230.375476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1230.394828] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1230.640240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.671445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.702425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.743073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.776563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.822662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.872254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.879274] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1230.920504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.937895] LNet: 66090:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.955732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.989446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.028086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.066369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.095170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.128533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.398320] LNet: 66133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.406265] LNetError: 66133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.422457] LNet: Removed LNI 192.168.203.6@tcp [ 1232.937736] Key type .llcrypt unregistered [ 1232.944071] Key type ._llcrypt unregistered [ 1253.107415] Key type ._llcrypt registered [ 1253.109506] Key type .llcrypt registered [ 1253.267878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.283497] alg: No test for adler32 (adler32-zlib) [ 1254.284729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1254.330803] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1263.210541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1273.005113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.046117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1273.080305] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1274.739654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e network_timeout -r 1 [ 1274.790797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e network_timeout -r 1 [ 1274.844402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e network_timeout -r 1 [ 1274.895250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e network_timeout -r 1 [ 1274.931730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e network_timeout -r 1 [ 1274.979822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e network_timeout -r 1 [ 1275.051410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e network_timeout -r 1 [ 1275.115220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e network_timeout -r 1 [ 1275.175598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1275.199239] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1275.390635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.413476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.442810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.495947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.540815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.592240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.644815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.681329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.703151] LNet: 67017:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.729094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.790235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.841195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.885331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.885654] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1275.914791] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.106@tcp [ 1275.938607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.983901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1281.401938] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1281.853866] LNet: 67096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.864617] LNetError: 67096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.872502] LNet: Removed LNI 192.168.203.6@tcp [ 1281.888030] LNet: Removed LNI 192.168.203.6@tcp1 [ 1282.332638] Key type .llcrypt unregistered [ 1282.335506] Key type ._llcrypt unregistered [ 1298.797627] Key type ._llcrypt registered [ 1298.799924] Key type .llcrypt registered [ 1299.934580] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:50:35 (1788202235) [ 1300.361571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1300.385802] alg: No test for adler32 (adler32-zlib) [ 1301.304323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.357367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1301.374890] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1302.192917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.219992] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1302.993943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 1303.083870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1303.157563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1303.218565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1303.280153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1303.331959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1303.390920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.6@tcp [ 1303.413645] LNetError: 68195:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.6@tcp to recovery [ 1304.415293] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1304.416483] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1306.463408] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1306.911970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1310.561523] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1311.540823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1311.576375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1311.644794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.671128] LNet: 68312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.680290] LNetError: 68312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.688249] LNet: Removed LNI 192.168.203.6@tcp [ 1311.698303] LNet: Removed LNI 192.168.203.6@tcp1 [ 1311.730414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1311.807306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1311.832165] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1312.557468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1312.578714] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1313.389652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 1313.436561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1313.488757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1313.530111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1313.569764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1313.607681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1313.648899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1313.677982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.6@tcp [ 1313.700617] LNetError: 68585:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.6@tcp to recovery [ 1313.823478] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1316.097747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1319.968359] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1319.977619] LNet: Skipped 1 previous similar message [ 1324.039362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1324.069711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1324.101439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.835549] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:51:07 (1788202267) [ 1331.950574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1331.981716] LNet: 69286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.996427] LNetError: 69286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.001185] LNetError: 69286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1332.020054] LNet: Removed LNI 192.168.203.6@tcp [ 1332.039637] LNet: Removed LNI 192.168.203.6@tcp1 [ 1332.084336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.152953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1332.195774] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1333.100727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1333.136635] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1334.154341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 1334.250335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1334.309841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1334.355268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1334.413317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1334.459578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1334.511358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.6@tcp --health 0 [ 1334.638512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1335.200729] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.6@tcp [ 1335.202368] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1335.214477] LNet: Skipped 1 previous similar message [ 1350.235782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.375634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1350.409805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1350.436881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.6@tcp --health 500 [ 1350.523716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.153776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1357.234123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.289402] LNet: 69785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.300254] LNetError: 69785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1357.314268] LNetError: 69785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1357.340082] LNet: Removed LNI 192.168.203.6@tcp [ 1357.373802] LNet: Removed LNI 192.168.203.6@tcp1 [ 1357.411759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1357.550979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1357.592083] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1358.707205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1358.732896] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1360.219781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 1360.295264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.396873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1360.446500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1360.509938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1360.584475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1360.636092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1360.691789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.6@tcp --health 0 [ 1361.635340] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1361.650763] LNet: Skipped 3 previous similar messages [ 1362.218287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.705902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1372.751179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1372.785371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1380.355632] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:51:55 (1788202315) [ 1380.906994] LNet: 70832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.917711] LNetError: 70832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.928260] LNetError: 70832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1380.933664] LNet: Removed LNI 192.168.203.6@tcp [ 1380.944916] LNet: Removed LNI 192.168.203.6@tcp1 [ 1381.663714] Key type .llcrypt unregistered [ 1381.668141] Key type ._llcrypt unregistered [ 1388.537667] Key type ._llcrypt registered [ 1388.541283] Key type .llcrypt registered [ 1388.798948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1388.820910] alg: No test for adler32 (adler32-zlib) [ 1389.766368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1389.800889] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1400.014781] Lustre: DEBUG MARKER: Initial discovery [ 1400.140917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.106@tcp [ 1403.125505] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1403.307205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1403.407992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.106@tcp [ 1403.459720] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1403.512307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1403.573494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.106@tcp -r 1 -m GET -l 3 [ 1403.608893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.106@tcp -r 1 -m GET -e local_error [ 1403.677315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.106@tcp -r 1 -m PUT -l 6 [ 1404.192392] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1405.493351] Lustre: DEBUG MARKER: Force oleg306-client.virtnet to discover 192.168.203.106@tcp (in background) [ 1405.620110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.106@tcp [ 1407.253380] Lustre: DEBUG MARKER: Force oleg306-server to discover 192.168.203.6@tcp [ 1408.735350] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1410.853700] Lustre: DEBUG MARKER: Wait for 71592 [ 1412.735919] Lustre: DEBUG MARKER: Finished wait on 71592 [ 1412.864693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1412.912653] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1412.927322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1413.344654] LNet: 72054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.349808] LNetError: 72054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.375362] LNet: Removed LNI 192.168.203.6@tcp [ 1414.011421] Key type .llcrypt unregistered [ 1414.014954] Key type ._llcrypt unregistered [ 1433.992387] Key type ._llcrypt registered [ 1433.995471] Key type .llcrypt registered [ 1435.256671] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:52:50 (1788202370) [ 1436.080650] Key type .llcrypt unregistered [ 1436.089693] Key type ._llcrypt unregistered [ 1442.701430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1442.899382] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1442.904596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1442.919256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1445.505556] Key type ._llcrypt registered [ 1445.513725] Key type .llcrypt registered [ 1445.763421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1445.778581] alg: No test for adler32 (adler32-zlib) [ 1446.738838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.806259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.832391] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1447.673330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1447.696950] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1456.368688] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:53:12 (1788202392) [ 1456.531469] LNet: 74135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.541571] LNetError: 74135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1456.559425] LNet: Removed LNI 10.1.2.3@tcp [ 1456.575785] LNet: Removed LNI 192.168.203.6@tcp [ 1457.281605] Key type .llcrypt unregistered [ 1457.289117] Key type ._llcrypt unregistered [ 1463.408076] Key type ._llcrypt registered [ 1463.412270] Key type .llcrypt registered [ 1463.594534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1463.609972] alg: No test for adler32 (adler32-zlib) [ 1464.484340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.526512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.563441] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1465.208549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1465.231270] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1466.542471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.6@tcp [ 1467.615524] LNetError: Unexpected error -22 connecting to 192.168.203.6@tcp at host 192.168.203.6:988 [ 1474.462540] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:53:29 (1788202409) [ 1474.604082] LNet: 75268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.619626] LNetError: 75268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1474.629577] LNet: Removed LNI 10.1.2.3@tcp [ 1474.643226] LNet: Removed LNI 192.168.203.6@tcp [ 1475.558738] Key type .llcrypt unregistered [ 1475.565219] Key type ._llcrypt unregistered [ 1485.065138] Key type ._llcrypt registered [ 1485.068429] Key type .llcrypt registered [ 1485.406781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1485.432800] alg: No test for adler32 (adler32-zlib) [ 1486.391660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.466482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.506144] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1487.349430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1487.368813] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 1488.217558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.6@tcp1 --nid 192.168.203.6@tcp2 [ 1492.654409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp1 192.168.203.6@tcp2 [ 1492.683174] LNetError: 76116:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.6@tcp1 to 192.168.203.6@tcp2 - Route table may be misconfigured [ 1492.717436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp2 192.168.203.6@tcp1 [ 1499.419620] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:53:54 (1788202434) [ 1499.674359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.705393] LNet: 76689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.713262] LNetError: 76689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.723171] LNet: Removed LNI 192.168.203.6@tcp1 [ 1500.773805] LNet: Removed LNI 192.168.203.6@tcp2 [ 1500.817295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.884186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1500.945519] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1501.749534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1501.773579] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1502.678680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 1502.757546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.6@tcp [ 1502.815453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp1 192.168.203.6@tcp1 [ 1502.873611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.6@tcp -d 192.168.203.6@tcp -e network_timeout [ 1502.968311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -e network_timeout [ 1503.006731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -e network_timeout [ 1503.033499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -e network_timeout [ 1503.058390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1503.071517] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1503.083474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1503.152959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1503.231476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1503.968517] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1509.852754] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:54:05 (1788202445) [ 1509.969319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.001748] LNet: 77545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.020334] LNetError: 77545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.027321] LNetError: 77545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1510.038679] LNet: Removed LNI 192.168.203.6@tcp [ 1511.086379] LNet: Removed LNI 192.168.203.6@tcp1 [ 1511.107591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.214266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1511.471758] LNet: 77593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.799370] Key type .llcrypt unregistered [ 1511.810423] Key type ._llcrypt unregistered [ 1524.375201] Key type ._llcrypt registered [ 1524.377476] Key type .llcrypt registered [ 1525.836246] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:54:21 (1788202461) [ 1526.222539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1526.242949] alg: No test for adler32 (adler32-zlib) [ 1527.173777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.834667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1530.599076] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:54:26 (1788202466) [ 1530.725145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.754731] LNet: 78686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.776491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.858116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.879817] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1531.690613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1531.731121] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1532.974874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1533.012338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp1 [ 1533.059426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp1 [ 1539.576331] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:54:35 (1788202475) [ 1541.258843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg306-server [ 1542.986192] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:54:38 (1788202478) [ 1545.078687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg306-server [ 1546.554492] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:54:42 (1788202482) [ 1548.656966] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg306-server [ 1550.299297] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:54:46 (1788202486) [ 1552.464464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg306-server [ 1554.237448] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:54:50 (1788202490) [ 1556.222163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg306-server [ 1557.749837] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:54:53 (1788202493) [ 1559.356730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg306-server [ 1561.090285] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:54:56 (1788202496) [ 1563.192555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg306-server [ 1565.225142] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:55:00 (1788202500) [ 1567.151670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg306-server [ 1569.011498] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:55:04 (1788202504) [ 1570.796463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg306-server [ 1573.036205] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:55:08 (1788202508) [ 1573.156670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.211416] LNet: 82563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.230146] LNetError: 82563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.244933] LNet: Removed LNI 192.168.203.6@tcp [ 1574.321918] LNet: Removed LNI 192.168.203.6@tcp1 [ 1574.396856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.528434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.565661] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1575.426445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1575.516425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1575.766390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.801513] LNet: 82717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.815601] LNetError: 82717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.824177] LNetError: 82717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1575.837399] LNet: Removed LNI 192.168.203.6@tcp [ 1575.864724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.927794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1575.973222] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1576.713166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1576.862291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.876563] LNetError: 82862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.894284] LNet: Removed LNI 192.168.203.6@tcp [ 1576.924540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.995738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1577.028675] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1577.665083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1577.764813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1577.929325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1577.966718] LNet: 83009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.974155] LNet: 83009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1579.056300] LNet: Removed LNI 192.168.203.6@tcp [ 1579.123129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.205748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1579.256821] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1580.278164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1580.497901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.527302] LNetError: 83155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.534695] LNetError: 83155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1581.632686] LNet: Removed LNI 192.168.203.6@tcp [ 1581.670494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.717291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.739684] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1582.499310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1582.571280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1582.829947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.848943] LNet: 83302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.854131] LNet: 83302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1583.914655] LNet: Removed LNI 192.168.203.6@tcp [ 1583.950175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.001695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1584.025411] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1584.624860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1584.797046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.849762] LNetError: 83447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.860870] LNetError: 83447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1584.879521] LNet: Removed LNI 192.168.203.6@tcp [ 1584.898635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.939777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1584.959518] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1585.621451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1585.720784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1585.945812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.050779] LNet: Removed LNI 192.168.203.6@tcp [ 1587.081787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.144378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1587.171777] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1587.909322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1588.106868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.165901] LNet: Removed LNI 192.168.203.6@tcp [ 1589.238553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.321676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.355262] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1590.113580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1590.209758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1590.550831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.601594] LNet: Removed LNI 192.168.203.6@tcp [ 1591.636094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.703404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1591.739478] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1592.562945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1593.013725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.051952] LNet: 84033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.065623] LNet: 84033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1593.090866] LNetError: 84033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.104926] LNetError: 84033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1593.142043] LNet: Removed LNI 192.168.203.6@tcp [ 1593.178700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.229048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.255035] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1593.926552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1594.025344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1594.327899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.382439] LNet: Removed LNI 192.168.203.6@tcp [ 1595.451172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.543285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1595.604254] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1596.519718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1596.869197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.931789] LNet: Removed LNI 192.168.203.6@tcp [ 1597.975575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.072816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1598.107919] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1599.039396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1599.111636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.186280] LNet: Removed LNI 192.168.203.6@tcp [ 1599.233788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.319823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.350872] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1600.229942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1600.310688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.357259] LNet: Removed LNI 192.168.203.6@tcp [ 1600.398903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.458690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1600.543943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.577670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.638117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1600.751894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1600.786970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.857818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1600.952218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.000206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.069261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1601.124922] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1601.991471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.028995] LNet: Removed LNI 192.168.203.6@tcp [ 1602.055522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.124793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1609.118472] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:55:44 (1788202544) [ 1609.348907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.373929] LNet: 85347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.381282] LNet: 85347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1609.406975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.460849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1609.492811] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/180] [ 1610.964518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.997154] LNetError: 85485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1611.000629] LNetError: 85485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1611.016746] LNet: Removed LNI 192.168.203.6@tcp231 [ 1611.040028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.127542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1611.153702] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/180] [ 1611.166486] LNet: 85493:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1611.793928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1611.848324] LNet: Removed LNI 192.168.203.6@tcp231 [ 1611.887842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1611.915951] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/180] [ 1613.335931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.380470] LNet: Removed LNI 192.168.203.6@tcp231 [ 1613.416974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.500623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1613.542131] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/0] [ 1613.552495] LNet: 85645:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1614.052277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1614.104353] LNet: Removed LNI 192.168.203.6@tcp231 [ 1614.135698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1614.176242] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/0] [ 1615.621050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.680946] LNet: Removed LNI 192.168.203.6@tcp231 [ 1615.720817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.859895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1615.887365] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/60] [ 1615.892537] LNet: 85797:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1616.474899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1616.521281] LNet: Removed LNI 192.168.203.6@tcp231 [ 1616.538027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1616.575665] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/60] [ 1617.846534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.889371] LNet: Removed LNI 192.168.203.6@tcp231 [ 1617.929335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.060977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1618.096934] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/180] [ 1618.113474] LNet: 85948:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1618.473455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1618.507548] LNet: Removed LNI 192.168.203.6@tcp231 [ 1618.527023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1618.557353] LNet: Added LNI 192.168.203.6@tcp231 [8/256/0/180] [ 1627.901488] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:56:03 (1788202563) [ 1628.039930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.099691] LNet: Removed LNI 192.168.203.6@tcp231 [ 1628.128925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.234708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1628.261815] LNet: Added LNI 192.168.203.6@tcp232 [8/256/0/180] [ 1629.031306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1629.098249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1629.137105] LNet: Removed LNI 192.168.203.6@tcp232 [ 1629.179758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1629.225674] LNet: Added LNI 192.168.203.6@tcp232 [8/256/0/180] [ 1630.018736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1630.076818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1630.113464] LNet: Removed LNI 192.168.203.6@tcp232 [ 1630.135329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1630.182961] LNet: Added LNI 192.168.203.6@tcp232 [8/256/0/180] [ 1630.809720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1630.852918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1630.883472] LNet: Removed LNI 192.168.203.6@tcp232 [ 1630.903105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1630.938483] LNet: Added LNI 192.168.203.6@tcp232 [8/256/0/180] [ 1631.745246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1631.793711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1631.832881] LNet: Removed LNI 192.168.203.6@tcp232 [ 1631.851627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1638.055729] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:56:13 (1788202573) [ 1639.621385] Key type .llcrypt unregistered [ 1639.624270] Key type ._llcrypt unregistered [ 1645.902405] Key type ._llcrypt registered [ 1645.904880] Key type .llcrypt registered [ 1646.121339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1646.143833] alg: No test for adler32 (adler32-zlib) [ 1647.097162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1647.133855] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1654.436501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1663.378285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.464457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1663.499149] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1665.264470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_interrupt -r 2 [ 1665.314469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_interrupt -r 2 [ 1665.384047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_interrupt -r 2 [ 1665.436781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_interrupt -r 2 [ 1665.482690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_interrupt -r 2 [ 1665.535539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_interrupt -r 2 [ 1665.576444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_interrupt -r 2 [ 1665.625756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_interrupt -r 2 [ 1665.672034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.106@tcp [ 1665.696806] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1665.708345] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1665.967863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1666.001563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1666.051380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1666.093913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1666.133308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1666.173638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1666.217941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1666.263604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1666.281882] LNet: 88454:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1666.299865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1666.329728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1666.363675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1666.394605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1666.429585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1666.470720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1666.964297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_dropped -r 2 [ 1667.012483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_dropped -r 2 [ 1667.047450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_dropped -r 2 [ 1667.078469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_dropped -r 2 [ 1667.119605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_dropped -r 2 [ 1667.160858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_dropped -r 2 [ 1667.196921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_dropped -r 2 [ 1667.230630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_dropped -r 2 [ 1667.273881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.106@tcp [ 1667.292986] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1667.534605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1667.560085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1667.592637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1667.630937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1667.661533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1667.689541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1667.714811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1667.742516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1667.770841] LNet: 88574:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1667.801320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1667.836413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1667.876439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1667.899440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1667.926288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1667.951373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1668.430473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_aborted -r 2 [ 1668.466564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_aborted -r 2 [ 1668.494408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_aborted -r 2 [ 1668.533689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_aborted -r 2 [ 1668.563475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_aborted -r 2 [ 1668.609355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_aborted -r 2 [ 1668.674455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_aborted -r 2 [ 1668.715913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_aborted -r 2 [ 1668.762151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.106@tcp [ 1668.797536] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1669.074238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1669.110672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1669.151324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1669.194326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1669.241921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1669.266906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1669.293879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1669.329516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1669.344595] LNet: 88695:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1669.371653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1669.412863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1669.463297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1669.497033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1669.536988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1669.591166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1670.194228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_no_route -r 2 [ 1670.235963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_no_route -r 2 [ 1670.273435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_no_route -r 2 [ 1670.324094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_no_route -r 2 [ 1670.363306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_no_route -r 2 [ 1670.398695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_no_route -r 2 [ 1670.441527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_no_route -r 2 [ 1670.480555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_no_route -r 2 [ 1670.515627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.106@tcp [ 1670.817085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1670.855197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1670.902822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1670.946692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1670.990817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1671.037757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1671.068685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1671.112217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1671.154216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1671.181177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1671.213652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1671.256799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1671.296838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1671.334605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1671.851201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e local_timeout -r 2 [ 1671.889679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e local_timeout -r 2 [ 1671.925636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e local_timeout -r 2 [ 1671.978526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e local_timeout -r 2 [ 1672.023223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e local_timeout -r 2 [ 1672.059831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e local_timeout -r 2 [ 1672.091198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e local_timeout -r 2 [ 1672.118217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e local_timeout -r 2 [ 1672.143692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.106@tcp [ 1672.163728] LNet: There was an unexpected network error while writing to 192.168.203.106: rc = -22 [ 1672.168123] LNet: Skipped 1 previous similar message [ 1672.463683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1672.504481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1672.545375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1672.576950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1672.606568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1672.657690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1672.713275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1672.761096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1672.788941] LNet: 88935:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1672.802432] LNet: 88935:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1672.823110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1672.856673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1672.900744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1672.940911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1672.962986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1672.999613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1673.504036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp -m GET -e remote_dropped -r 2 [ 1673.565340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp -d 192.168.203.106@tcp1 -m GET -e remote_dropped -r 2 [ 1673.612333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp -m GET -e remote_dropped -r 2 [ 1673.645925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp -d 192.168.203.6@tcp1 -m GET -e remote_dropped -r 2 [ 1673.673347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp -m GET -e remote_dropped -r 2 [ 1673.721145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.106@tcp1 -m GET -e remote_dropped -r 2 [ 1673.780452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp -m GET -e remote_dropped -r 2 [ 1673.820029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.6@tcp1 -d 192.168.203.6@tcp1 -m GET -e remote_dropped -r 2 [ 1673.864496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.6@tcp 192.168.203.106@tcp [ 1673.891990] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.106@tcp [ 1674.090560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1674.137462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1674.165726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1674.186590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1674.216633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1674.242191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1674.274733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1674.312449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1674.347315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1674.376422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1674.406565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1674.439634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1674.469197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1674.497648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1679.337277] LNet: 89133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.344617] LNetError: 89133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.351058] LNet: Removed LNI 192.168.203.6@tcp [ 1679.362976] LNet: Removed LNI 192.168.203.6@tcp1 [ 1679.801286] Key type .llcrypt unregistered [ 1679.810234] Key type ._llcrypt unregistered [ 1695.552534] Key type ._llcrypt registered [ 1695.557153] Key type .llcrypt registered [ 1696.675282] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:57:12 (1788202632) [ 1697.215901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1697.227897] alg: No test for adler32 (adler32-zlib) [ 1698.124815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.188373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.216306] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1699.060594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1699.119877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.6@tcp [ 1699.151629] LNetError: 90112:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.6@tcp to recovery [ 1699.168302] LNet: 1 local NIs in recovery (showing 1): 192.168.203.6@tcp [ 1699.249732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.6@tcp [ 1699.399430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1705.381607] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:57:21 (1788202641) [ 1705.643807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.667220] LNet: 90704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.674329] LNetError: 90704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.727571] LNet: Removed LNI 192.168.203.6@tcp [ 1706.792886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.878680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.904929] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1707.606439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1707.623418] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 1708.642198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.6@tcp [ 1708.672295] LNetError: 90970:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.6@tcp to recovery [ 1708.897459] LNet: There was an unexpected network error while writing to 192.168.203.6: rc = -22 [ 1709.978507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.6@tcp [ 1716.836216] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:57:32 (1788202652) [ 1718.656568] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg306-server [ 1720.481080] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:57:36 (1788202656) [ 1722.188089] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg306-server [ 1723.919312] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:57:39 (1788202659) [ 1725.646426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg306-server [ 1727.163600] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:57:43 (1788202663) [ 1727.369078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1727.398400] LNet: 92591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1727.407434] LNetError: 92591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1727.418578] LNet: Removed LNI 192.168.203.6@tcp [ 1727.440159] LNet: Removed LNI 192.168.203.6@tcp1 [ 1727.469487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.545491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.573384] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1728.202363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1728.286380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1728.353240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1728.437182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1728.491173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1728.536853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1734.352638] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:57:50 (1788202670) [ 1734.564324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.591887] LNet: 93316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.602774] LNetError: 93316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.614291] LNetError: 93316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1734.629715] LNet: Removed LNI 192.168.203.6@tcp [ 1734.665725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.743863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.777159] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1741.659604] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:57:57 (1788202677) [ 1741.796184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.860901] LNet: 94017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1741.871600] LNetError: 94017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.892437] LNet: Removed LNI 192.168.203.6@tcp [ 1741.957307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.045546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1742.076468] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1750.102372] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:58:05 (1788202685) [ 1751.963779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1753.693533] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:58:09 (1788202689) [ 1754.222506] LNet: 95054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.234115] LNetError: 95054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.247694] LNet: Removed LNI 192.168.203.6@tcp [ 1754.824635] Key type .llcrypt unregistered [ 1754.829839] Key type ._llcrypt unregistered [ 1760.387464] Key type ._llcrypt registered [ 1760.389584] Key type .llcrypt registered [ 1760.632524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1760.662858] alg: No test for adler32 (adler32-zlib) [ 1761.607483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1761.682052] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1769.780083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 1776.567657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.106@tcp [ 1777.007254] LNet: 95481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.027934] LNetError: 95481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.059396] LNet: Removed LNI 192.168.203.6@tcp [ 1777.592727] Key type .llcrypt unregistered [ 1777.603524] Key type ._llcrypt unregistered [ 1789.687781] Key type ._llcrypt registered [ 1789.695318] Key type .llcrypt registered [ 1791.058743] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1792.561634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1794.202930] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:58:50 (1788202730) [ 1795.671397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1797.566602] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:58:53 (1788202733) [ 1799.569772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg306-server [ 1800.982426] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:58:56 (1788202736) [ 1802.714914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg306-server [ 1804.467314] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:59:00 (1788202740) [ 1805.505804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1805.715260] alg: No test for adler32 (adler32-zlib) [ 1806.730437] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 1806.930313] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1808.615465] Key type lgssc registered [ 1809.824645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.398593] Key type lgssc unregistered [ 1830.679735] LNet: 98660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.686162] LNetError: 98660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.700446] LNet: Removed LNI 192.168.203.6@tcp [ 1831.345977] Key type .llcrypt unregistered [ 1831.347503] Key type ._llcrypt unregistered [ 1845.345689] Key type ._llcrypt registered [ 1845.349427] Key type .llcrypt registered [ 1846.789513] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:59:42 (1788202782) [ 1847.238923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.251693] alg: No test for adler32 (adler32-zlib) [ 1848.220781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.281429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1848.342915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1848.421111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1848.486447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1855.261603] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:59:50 (1788202790) [ 1855.497813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.539049] LNet: 100076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.577503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.626991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.654637] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1856.487587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1856.520403] LNet: 100220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1856.530703] LNetError: 100220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1856.544300] LNet: Removed LNI 192.168.203.6@tcp [ 1856.572602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.623729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1856.647570] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1857.087178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1857.108682] LNetError: 100248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.132862] LNet: Removed LNI 192.168.203.6@tcp [ 1857.160346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.248179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1857.300283] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1864.055664] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:59:59 (1788202799) [ 1864.284932] LNet: 100838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.290604] LNet: 100838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1864.297399] LNetError: 100838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.312728] LNet: Removed LNI 192.168.203.6@tcp [ 1865.075546] Key type .llcrypt unregistered [ 1865.078615] Key type ._llcrypt unregistered [ 1871.260497] Key type ._llcrypt registered [ 1871.269227] Key type .llcrypt registered [ 1871.466481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1871.484955] alg: No test for adler32 (adler32-zlib) [ 1872.421301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.519215] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1872.528502] LNet: *** cfs_fail_loc=e003, val=100*** [ 1873.316560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1873.338144] LNet: *** cfs_fail_loc=e003, val=100*** [ 1873.340837] LNet: Skipped 2 previous similar messages [ 1873.349563] Ping reply received with mapping: [ 1873.352454] found metadata reply NI[192.168.203.6@tcp] [ 1873.355917] 06 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1873.358960] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1873.362735] 5a 5a 5a 5a 5a 5a 5a 5a [ 1873.500473] LNet: 101239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.508810] LNetError: 101239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.524229] LNet: Removed LNI 192.168.203.6@tcp [ 1874.158802] Key type .llcrypt unregistered [ 1874.164110] Key type ._llcrypt unregistered [ 1884.861856] Key type ._llcrypt registered [ 1884.864698] Key type .llcrypt registered [ 1886.150897] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:00:21 (1788202821) [ 1886.729524] Key type .llcrypt unregistered [ 1886.735869] Key type ._llcrypt unregistered [ 1892.651683] Key type ._llcrypt registered [ 1892.654412] Key type .llcrypt registered [ 1892.820201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1892.845954] alg: No test for adler32 (adler32-zlib) [ 1893.978672] LNetError: 102299:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1894.408707] Key type .llcrypt unregistered [ 1894.421415] Key type ._llcrypt unregistered [ 1899.750246] Key type ._llcrypt registered [ 1899.752453] Key type .llcrypt registered [ 1901.031234] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:00:36 (1788202836) [ 1901.764056] Key type .llcrypt unregistered [ 1901.766925] Key type ._llcrypt unregistered [ 1908.639996] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1908.645769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1911.177316] Key type ._llcrypt registered [ 1911.179223] Key type .llcrypt registered [ 1911.373167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1911.391928] alg: No test for adler32 (adler32-zlib) [ 1912.385659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.481061] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1922.025531] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:00:58 (1788202858) [ 1922.426766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.462971] LNet: 104088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.475487] LNetError: 104088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1922.500749] LNet: Removed LNI 10.1.2.3@tcp [ 1922.550605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.878639] Key type .llcrypt unregistered [ 1923.884636] Key type ._llcrypt unregistered [ 1931.050778] Key type ._llcrypt registered [ 1931.061947] Key type .llcrypt registered [ 1931.340680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1931.362740] alg: No test for adler32 (adler32-zlib) [ 1932.240728] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1932.308769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1933.615742] LNet: 104402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.624462] LNetError: 104402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.640173] LNet: Removed LNI 192.168.203.6@tcp [ 1934.316517] Key type .llcrypt unregistered [ 1934.324029] Key type ._llcrypt unregistered [ 1948.675240] Key type ._llcrypt registered [ 1948.679087] Key type .llcrypt registered [ 1950.036368] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:01:25 (1788202885) [ 1950.903375] Key type .llcrypt unregistered [ 1950.911487] Key type ._llcrypt unregistered [ 1957.663485] Key type ._llcrypt registered [ 1957.665715] Key type .llcrypt registered [ 1957.955576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1957.979924] alg: No test for adler32 (adler32-zlib) [ 1965.135927] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:01:40 (1788202900) [ 1965.266769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.321269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.401951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.441265] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1966.290984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.326833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1966.383506] LNet: Removed LNI 192.168.203.6@tcp [ 1966.609678] LNet: 106158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.985447] Key type .llcrypt unregistered [ 1966.987671] Key type ._llcrypt unregistered [ 1978.226920] Key type ._llcrypt registered [ 1978.229674] Key type .llcrypt registered [ 1979.692127] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:01:55 (1788202915) [ 1980.141578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.166964] alg: No test for adler32 (adler32-zlib) [ 1981.065944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.110820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.133698] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 1981.821754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 1981.862266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.6@tcp [ 1981.881789] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1981.883786] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.895072] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 0 [ 1981.906779] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1981.911748] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1981.917335] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.920657] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 0 [ 1981.928990] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1981.933868] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1981.946562] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.949941] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 2031 [ 1981.960772] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1981.965758] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1981.971519] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1981.975929] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1981, 0, 2031 [ 1981.981974] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1988.405285] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:02:04 (1788202924) [ 1989.016826] LNet: 107697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1989.028534] LNetError: 107697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1989.043422] LNet: Removed LNI 192.168.203.6@tcp [ 1989.735370] Key type .llcrypt unregistered [ 1989.738990] Key type ._llcrypt unregistered [ 1996.239800] Key type ._llcrypt registered [ 1996.246898] Key type .llcrypt registered [ 1996.563073] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.592817] alg: No test for adler32 (adler32-zlib) [ 1997.552594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1997.587829] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2004.985928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 2015.465181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.530876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2015.542908] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 2016.477321] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2016.635183] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2016.638592] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2016.643500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2018.990475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2019.023676] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2030.526848] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:02:46 (1788202966) [ 2030.811925] LNet: 109207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.818440] LNetError: 109207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.831599] LNet: Removed LNI 192.168.203.6@tcp [ 2031.839172] LNet: Removed LNI 192.168.203.6@tcp1 [ 2031.841844] LNetError: 109207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2031.848311] LNetError: 109207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2031.861521] LNet: Removed LNI 10.1.2.3@tcp99 [ 2032.825564] Key type .llcrypt unregistered [ 2032.827101] Key type ._llcrypt unregistered [ 2039.097793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2039.102519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2041.664411] Key type ._llcrypt registered [ 2041.666431] Key type .llcrypt registered [ 2041.821487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2041.830661] alg: No test for adler32 (adler32-zlib) [ 2042.736606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2042.829369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2042.860092] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2043.530254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2043.574705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2046.102978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2046.161883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2046.217959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2046.256523] LNetError: 109956:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2046.314785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2046.378909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2046.875462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2046.921467] LNet: 109976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.926975] LNetError: 109976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2046.933168] LNet: Removed LNI 10.1.2.3@tcp [ 2047.987394] LNet: Removed LNI 192.168.203.6@tcp [ 2048.730886] Key type .llcrypt unregistered [ 2048.733335] Key type ._llcrypt unregistered [ 2059.794323] Key type ._llcrypt registered [ 2059.798192] Key type .llcrypt registered [ 2061.077915] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:03:16 (1788202996) [ 2061.834569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.858954] alg: No test for adler32 (adler32-zlib) [ 2062.784574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.830836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2062.851895] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2063.515657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2063.620484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg306-client@tcp [ 2069.692845] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:03:25 (1788203005) [ 2069.782962] LNet: 111505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.793031] LNetError: 111505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.828303] LNet: Removed LNI 192.168.203.6@tcp [ 2071.512602] Key type .llcrypt unregistered [ 2071.518099] Key type ._llcrypt unregistered [ 2077.211323] Key type ._llcrypt registered [ 2077.217201] Key type .llcrypt registered [ 2077.370034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2077.402973] alg: No test for adler32 (adler32-zlib) [ 2078.294415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2078.315788] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2078.342197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2084.037775] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:03:40 (1788203020) [ 2085.725783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2087.282547] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:03:43 (1788203023) [ 2088.805665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2090.340499] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:03:46 (1788203026) [ 2090.772429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.799539] LNet: 112935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.811318] LNetError: 112935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.828742] LNet: Removed LNI 192.168.203.6@tcp [ 2090.858512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.927602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2090.958774] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2091.789121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2091.848513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2098.677318] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:03:54 (1788203034) [ 2099.055138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.093592] LNet: 113644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2099.104726] LNetError: 113644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2099.126913] LNet: Removed LNI 192.168.203.6@tcp [ 2099.169366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.239465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2099.295801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2099.979714] Key type .llcrypt unregistered [ 2099.984770] Key type ._llcrypt unregistered [ 2110.797213] Key type ._llcrypt registered [ 2110.799324] Key type .llcrypt registered [ 2112.355480] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:04:07 (1788203047) [ 2112.929735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2112.961793] alg: No test for adler32 (adler32-zlib) [ 2113.919547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.968484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.998664] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2114.682476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2114.766493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2120.568668] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:04:16 (1788203056) [ 2120.900463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.925882] LNet: 115204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.938623] LNetError: 115204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.968319] LNet: Removed LNI 192.168.203.6@tcp [ 2120.989823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.045391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2121.079595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2126.381387] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:04:22 (1788203062) [ 2126.494668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.515124] LNet: 115777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.573231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.647258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2127.009718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2127.052123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2127.135601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2127.608418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2127.642319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2127.710459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.726934] LNet: 115817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.759548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.807939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2128.166808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.204908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.278783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2128.821773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.866373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2134.644788] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:04:30 (1788203070) [ 2134.784479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.830597] LNet: 116415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.867702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.915695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.964365] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2135.764621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.788836] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2136.862666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2136.944725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2137.191270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.234354] LNetError: 116696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.244852] LNet: Removed LNI 192.168.203.6@tcp [ 2137.258292] LNet: Removed LNI 192.168.203.6@tcp2 [ 2137.280493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.349697] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2137.358795] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2137.365766] LNet: 116702:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2138.030422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2138.265773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.287889] LNetError: 116739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.293681] LNetError: 116739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2139.359685] LNet: Removed LNI 192.168.203.6@tcp [ 2139.376935] LNet: Removed LNI 192.168.203.6@tcp2 [ 2139.409711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.490697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.518922] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2140.173951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2140.210147] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2141.092744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2141.136077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2141.185943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2141.389068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.419150] LNet: 117024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.428143] LNet: 117024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2141.440394] LNetError: 117024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.446801] LNetError: 117024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2141.454412] LNet: Removed LNI 192.168.203.6@tcp [ 2142.511048] LNet: Removed LNI 192.168.203.6@tcp2 [ 2142.536286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.603910] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2142.612703] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2142.617922] LNet: 117030:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2143.231883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2143.417460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.448770] LNet: Removed LNI 192.168.203.6@tcp [ 2144.486584] LNet: Removed LNI 192.168.203.6@tcp2 [ 2144.516432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.555214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.597443] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2145.206591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2145.224033] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2145.935865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2145.999637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2146.175810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.198509] LNetError: 117347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.206632] LNetError: 117347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2146.212722] LNet: Removed LNI 192.168.203.6@tcp [ 2146.228652] LNet: Removed LNI 192.168.203.6@tcp2 [ 2146.243801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.291746] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2146.298724] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2146.303782] LNet: 117353:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2146.726055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2146.922385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.999139] LNet: Removed LNI 192.168.203.6@tcp [ 2148.007598] LNet: Removed LNI 192.168.203.6@tcp2 [ 2148.027356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.077072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.098927] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2148.780086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2148.800529] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2149.561512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2149.600732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2149.660176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2149.831673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.859727] LNet: 117674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.868237] LNet: 117674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2150.943280] LNet: Removed LNI 192.168.203.6@tcp [ 2151.975641] LNet: Removed LNI 192.168.203.6@tcp2 [ 2152.002091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.068298] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2152.074608] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2152.078204] LNet: 117681:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2152.391629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2152.571313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.584447] LNet: Removed LNI 192.168.203.6@tcp [ 2153.639510] LNet: Removed LNI 192.168.203.6@tcp2 [ 2153.680354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.725142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.749614] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2154.575573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2154.609141] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2155.348426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2155.378219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2155.444614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2155.836356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.867925] LNetError: 118015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.881107] LNetError: 118015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2155.889344] LNet: Removed LNI 192.168.203.6@tcp [ 2155.903450] LNet: Removed LNI 192.168.203.6@tcp2 [ 2155.924543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.007882] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2156.017552] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2156.708947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2157.044814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.111157] LNet: Removed LNI 192.168.203.6@tcp [ 2158.119586] LNet: Removed LNI 192.168.203.6@tcp2 [ 2158.181077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.263989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.294627] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2159.012134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2159.039566] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2159.929978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2159.974228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2160.022916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2160.078795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2160.400567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.439262] LNet: Removed LNI 192.168.203.6@tcp [ 2162.473415] LNet: Removed LNI 192.168.203.6@tcp2 [ 2162.510636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.593750] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2162.609428] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2162.615656] LNet: 118381:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2162.621921] LNet: 118381:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 2163.051981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2169.592050] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2171.273515] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:05:07 (1788203107) [ 2171.539967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.567920] LNet: 119141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.574154] LNet: 119141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2171.583787] LNet: Removed LNI 192.168.203.6@tcp [ 2172.650791] LNet: Removed LNI 192.168.203.6@tcp2 [ 2172.682122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.987843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.004061] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2173.780461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2173.794468] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2174.453941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2174.466623] LNetError: 119414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2174.471053] LNetError: 119414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2174.476100] LNet: Removed LNI 10.1.2.3@tcp [ 2174.491305] LNet: Removed LNI 192.168.203.6@tcp [ 2174.517468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.578275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2174.604840] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2174.610785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2175.160814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2175.201200] LNet: Removed LNI 192.168.203.6@tcp [ 2175.216831] LNet: Removed LNI 10.1.2.3@tcp [ 2181.304338] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:05:17 (1788203117) [ 2181.473529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.542599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.603488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.631993] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2182.300474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2182.343684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.378377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.433907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.474281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.516523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.555276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.604737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.656219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.710594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.743465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.6@tcp [ 2182.872593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2188.754027] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:05:24 (1788203124) [ 2188.981094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2190.057717] LNet: Removed LNI 192.168.203.6@tcp [ 2190.111514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2190.266435] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2190.408625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2190.416794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2190.429717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2192.794165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2192.824150] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2193.498900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2193.519456] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2195.708765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2195.752545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2195.830254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.6@tcp -m PUT -e local_timeout -r 1 [ 2195.912382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2196.003719] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2196.013345] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2196.279831] LNetError: 120820:0:(lib-move.c:4801:lnet_parse()) 192.168.203.6@tcp, src 192.168.203.6@tcp: Dropping PUT (error -108 looking up sender) [ 2197.285708] LNet: Removed LNI 10.1.2.3@tcp [ 2198.310263] LNet: Removed LNI 192.168.203.6@tcp [ 2198.712793] Key type .llcrypt unregistered [ 2198.716246] Key type ._llcrypt unregistered [ 2210.860477] Key type ._llcrypt registered [ 2210.862319] Key type .llcrypt registered [ 2212.378719] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:05:47 (1788203147) [ 2212.703568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.734765] alg: No test for adler32 (adler32-zlib) [ 2213.739762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.345225] LNet: 122137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.035494] Key type .llcrypt unregistered [ 2215.039502] Key type ._llcrypt unregistered [ 2220.931475] Key type ._llcrypt registered [ 2220.937032] Key type .llcrypt registered [ 2221.232169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2221.254961] alg: No test for adler32 (adler32-zlib) [ 2222.278613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2222.323348] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2230.872759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 2231.170820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2232.001139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.106@tcp [ 2232.263628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2232.314589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2232.373562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2232.447275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2232.517682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2232.575687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2232.607807] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.106@tcp [ 2232.614685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2232.651182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2232.679925] LNet: 122591:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2232.701992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2232.747428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2232.783463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2232.813537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2232.864623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2232.900536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2233.702098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2233.972243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2233.996905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2234.025931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2234.048413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2234.070428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2234.100189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2234.123281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2234.147096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2234.164216] LNet: 122709:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2234.179213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2234.205564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2234.226807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2234.259540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2234.305760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2234.355275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2235.054551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.106@tcp [ 2235.347078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2235.404399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2235.444986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2235.484023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2235.534550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2235.582673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2235.620855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2235.663677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2235.696195] LNet: 122826:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2235.715481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2235.768087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2235.822110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2235.893136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2235.946745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2235.993445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2236.883444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2237.139406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2237.167503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2237.191423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2237.229265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2237.274743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2237.321868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2237.363347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2237.422983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2237.476841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2237.518439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2237.587743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2237.646016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2237.733158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2237.803389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2238.401491] LNet: 122993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.408923] LNetError: 122993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2238.441724] LNet: Removed LNI 192.168.203.6@tcp [ 2238.877894] Key type .llcrypt unregistered [ 2238.880784] Key type ._llcrypt unregistered [ 2257.460467] Key type ._llcrypt registered [ 2257.465286] Key type .llcrypt registered [ 2258.958338] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:06:34 (1788203194) [ 2259.444913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2259.468445] alg: No test for adler32 (adler32-zlib) [ 2260.378693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.798169] LNet: 123881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2261.472365] Key type .llcrypt unregistered [ 2261.474316] Key type ._llcrypt unregistered [ 2267.672377] Key type ._llcrypt registered [ 2267.678578] Key type .llcrypt registered [ 2267.919641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2267.935867] alg: No test for adler32 (adler32-zlib) [ 2268.829620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2268.858228] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2277.156603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.106@tcp [ 2286.290216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.327584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2286.350387] LNet: Added LNI 192.168.203.6@tcp1 [8/256/0/180] [ 2287.132714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2287.768369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.106@tcp [ 2288.036690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.063624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2288.090353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.114194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.155908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.210918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.255453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.299504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.317533] LNet: 124538:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.342293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.351265] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.106@tcp [ 2288.380717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.420695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.463313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.498039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.536200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2289.224285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2289.507432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.527885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2289.545621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2289.579387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2289.618173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.650256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.709300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.761294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2289.805227] LNet: 124656:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2289.839828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2289.886549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2289.942904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2289.970489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2290.016086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2290.085761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.873625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.106@tcp [ 2291.218426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2291.273500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2291.320542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2291.350873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2291.374677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.424692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.473744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.517883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.541684] LNet: 124773:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.561592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.589762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.625807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.658891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.712085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2291.747794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2292.347221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2292.616659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2292.668324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2292.701990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2292.738445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2292.763252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2292.802948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2292.840495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2292.886440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2292.922217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2292.981403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2293.031131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2293.084743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2293.136549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2293.182446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2298.816816] LNet: 124975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2298.825791] LNetError: 124975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2298.836463] LNet: Removed LNI 192.168.203.6@tcp [ 2298.850678] LNet: Removed LNI 192.168.203.6@tcp1 [ 2299.203859] Key type .llcrypt unregistered [ 2299.207054] Key type ._llcrypt unregistered [ 2315.943969] Key type ._llcrypt registered [ 2315.952361] Key type .llcrypt registered [ 2317.401319] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:07:32 (1788203252) [ 2317.992777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2318.019192] alg: No test for adler32 (adler32-zlib) [ 2319.046800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2319.114902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.144357] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2320.055689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2320.080545] LNet: Added LNI 192.168.203.6@tcp2 [8/256/0/180] [ 2321.220396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.6@tcp [ 2321.285556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2322.335160] LNet: Removed LNI 192.168.203.6@tcp [ 2329.765692] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:07:45 (1788203265) [ 2331.226180] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2333.819735] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:07:49 (1788203269) [ 2340.813503] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:07:56 (1788203276) [ 2347.059714] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:08:03 (1788203283) [ 2354.624398] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:08:10 (1788203290) [ 2361.871381] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:08:17 (1788203297) [ 2370.003575] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:08:25 (1788203305) [ 2377.806550] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:08:33 (1788203313) [ 2385.314466] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:08:41 (1788203321) [ 2393.910929] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:08:49 (1788203329) [ 2402.105366] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:08:57 (1788203337) [ 2410.813530] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:09:06 (1788203346) [ 2418.352703] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:09:13 (1788203353) [ 2427.404795] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 15:09:22 (1788203362) [ 2436.479528] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 15:09:31 (1788203371) [ 2444.606319] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 15:09:40 (1788203380) [ 2451.544321] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 15:09:47 (1788203387) [ 2459.188402] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 15:09:54 (1788203394) [ 2466.734955] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 15:10:02 (1788203402) [ 2466.999462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.042392] LNet: 136676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.058029] LNetError: 136676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.142558] LNet: Removed LNI 192.168.203.6@tcp2 [ 2468.185547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.312781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2468.333249] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2469.499455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2469.523529] LNet: 136818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.531584] LNetError: 136818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.551288] LNet: Removed LNI 192.168.203.6@tcp [ 2469.579376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2469.655921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2469.682741] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2476.869550] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 15:10:12 (1788203412) [ 2483.884605] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 15:10:19 (1788203419) [ 2491.169840] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 15:10:27 (1788203427) [ 2498.330333] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 15:10:34 (1788203434) [ 2506.519470] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 15:10:42 (1788203442) [ 2515.469570] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 15:10:50 (1788203450) [ 2523.092550] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 15:10:58 (1788203458) [ 2530.979940] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 15:11:06 (1788203466) [ 2540.480551] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 15:11:15 (1788203475) [ 2550.546963] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 15:11:26 (1788203486) [ 2558.963465] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 15:11:34 (1788203494) [ 2566.589445] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 15:11:41 (1788203501) [ 2573.359107] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2357 sec ========== 15:11:49 (1788203509) [ 2573.647356] LNet: 144283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.653983] LNetError: 144283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2573.671775] LNet: Removed LNI 192.168.203.6@tcp [ 2574.532875] Key type .llcrypt unregistered [ 2574.535791] Key type ._llcrypt unregistered [ 2587.808905] Key type ._llcrypt registered [ 2587.812562] Key type .llcrypt registered [ 2588.318562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2588.334767] alg: No test for adler32 (adler32-zlib) [ 2589.350902] Lustre: Lustre: Build Version: 2.17.56_54_gb664824 [ 2589.545204] LNet: Added LNI 192.168.203.6@tcp [8/256/0/180] [ 2591.271512] Key type lgssc registered [ 2592.651977] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2668.199471] Lustre: Mounted lustre-client - version 2.17.56_54_gb664824 [ 2673.902807] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2682.170776] Lustre: Unmounted lustre-client [ 2721.798317] Key type lgssc unregistered [ 2722.203996] LNet: 147248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2722.213919] LNetError: 147248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.250207] LNet: Removed LNI 192.168.203.6@tcp [ 2723.328908] Key type .llcrypt unregistered [ 2723.335073] Key type ._llcrypt unregistered