[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 391889689 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003248] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.007930] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010126] LSM: Security Framework initializing [ 0.011033] Yama: becoming mindful. [ 0.012020] SELinux: Initializing. [ 0.013037] *** VALIDATE selinux *** [ 0.020212] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023224] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026074] *** VALIDATE tmpfs *** [ 0.028037] *** VALIDATE proc *** [ 0.028827] *** VALIDATE cgroup *** [ 0.029004] *** VALIDATE cgroup2 *** [ 0.030163] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031125] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033024] Spectre V2 : User space: Vulnerable [ 0.033835] Speculative Store Bypass: Vulnerable [ 0.036270] debug: unmapping init [mem 0xffffffff99259000-0xffffffff99260fff] [ 0.038905] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039534] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040016] ... version: 2 [ 0.040884] ... bit width: 48 [ 0.041007] ... generic registers: 4 [ 0.041835] ... value mask: 0000ffffffffffff [ 0.042008] ... max period: 00007fffffffffff [ 0.043007] ... fixed-purpose events: 3 [ 0.044006] ... event mask: 000000070000000f [ 0.045263] rcu: Hierarchical SRCU implementation. [ 0.047289] smp: Bringing up secondary CPUs ... [ 0.048483] x86: Booting SMP configuration: [ 0.049022] .... node #0, CPUs: #1 #2 #3 [ 0.057254] smp: Brought up 1 node, 4 CPUs [ 0.059008] smpboot: Max logical packages: 1 [ 0.059926] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.131170] node 0 deferred pages initialised in 71ms [ 0.134121] devtmpfs: initialized [ 0.135185] x86/mm: Memory block size: 128MB [ 0.137081] gcov: version magic: 0x41383552 [ 0.138511] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141053] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.144007] pinctrl core: initialized pinctrl subsystem [ 0.145081] [ 0.145445] ************************************************************* [ 0.147006] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.148005] ** ** [ 0.149005] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151006] ** ** [ 0.153008] ** This means that this kernel is built to expose internal ** [ 0.155007] ** IOMMU data structures, which may compromise security on ** [ 0.156005] ** your system. ** [ 0.158006] ** ** [ 0.159005] ** If you see this message and you are not debugging the ** [ 0.160006] ** kernel, report this immediately to your vendor! ** [ 0.162006] ** ** [ 0.163005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165007] ************************************************************* [ 0.166665] NET: Registered protocol family 16 [ 0.167500] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170034] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.172033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175085] cpuidle: using governor menu [ 0.177025] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179030] PCI: Using configuration type 1 for base access [ 0.182118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189044] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.191043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.194169] cryptd: max_cpu_qlen set to 1000 [ 0.196215] ACPI: Added _OSI(Module Device) [ 0.197010] ACPI: Added _OSI(Processor Device) [ 0.198007] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.199048] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202675] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207380] ACPI: Interpreter enabled [ 0.208037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211012] ACPI: Using IOAPIC for interrupt routing [ 0.212102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214334] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.224462] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230038] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.238148] acpiphp: Slot [2] registered [ 0.239064] acpiphp: Slot [5] registered [ 0.239900] acpiphp: Slot [6] registered [ 0.241104] acpiphp: Slot [3] registered [ 0.243072] acpiphp: Slot [4] registered [ 0.244071] acpiphp: Slot [7] registered [ 0.245069] acpiphp: Slot [8] registered [ 0.246068] acpiphp: Slot [9] registered [ 0.247053] acpiphp: Slot [10] registered [ 0.247926] acpiphp: Slot [11] registered [ 0.249048] acpiphp: Slot [12] registered [ 0.250060] acpiphp: Slot [13] registered [ 0.251082] acpiphp: Slot [14] registered [ 0.252065] acpiphp: Slot [15] registered [ 0.254094] acpiphp: Slot [16] registered [ 0.255092] acpiphp: Slot [17] registered [ 0.256063] acpiphp: Slot [18] registered [ 0.257075] acpiphp: Slot [19] registered [ 0.259054] acpiphp: Slot [20] registered [ 0.260035] acpiphp: Slot [21] registered [ 0.261051] acpiphp: Slot [22] registered [ 0.262065] acpiphp: Slot [23] registered [ 0.263082] acpiphp: Slot [24] registered [ 0.265078] acpiphp: Slot [25] registered [ 0.266091] acpiphp: Slot [26] registered [ 0.267150] acpiphp: Slot [27] registered [ 0.269074] acpiphp: Slot [28] registered [ 0.270068] acpiphp: Slot [29] registered [ 0.272149] acpiphp: Slot [30] registered [ 0.272964] acpiphp: Slot [31] registered [ 0.273037] PCI host bridge to bus 0000:00 [ 0.273900] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.275009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.276008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.278008] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.279009] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.281012] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.282117] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283650] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285826] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.290342] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.292368] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.294008] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.296035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.297010] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.299437] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.300000] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.303040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.305858] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.311013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.321755] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.325009] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.329408] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.335016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.340009] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.359013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.367761] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.373014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.378011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.389013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.398120] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.400496] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.402303] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.405225] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.407157] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.411275] iommu: Default domain type: Passthrough [ 0.412352] SCSI subsystem initialized [ 0.413148] ACPI: bus type USB registered [ 0.414084] usbcore: registered new interface driver usbfs [ 0.416054] usbcore: registered new interface driver hub [ 0.418084] usbcore: registered new device driver usb [ 0.419114] pps_core: LinuxPPS API ver. 1 registered [ 0.420007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.422032] PTP clock support registered [ 0.423114] EDAC MC: Ver: 3.0.0 [ 0.425141] PCI: Using ACPI for IRQ routing [ 0.426672] NetLabel: Initializing [ 0.428007] NetLabel: domain hash size = 128 [ 0.430008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.431064] NetLabel: unlabeled traffic allowed by default [ 0.433115] vgaarb: loaded [ 0.434221] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.436006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.442004] clocksource: Switched to clocksource kvm-clock [ 0.539340] VFS: Disk quotas dquot_6.6.0 [ 0.540891] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.543405] *** VALIDATE ramfs *** [ 0.544726] *** VALIDATE hugetlbfs *** [ 0.546206] pnp: PnP ACPI init [ 0.548759] pnp: PnP ACPI: found 6 devices [ 0.565776] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.569126] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.571629] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.573869] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.575743] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.577774] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.581214] NET: Registered protocol family 2 [ 0.583543] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.590766] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.593690] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.597218] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.599649] TCP: Hash tables configured (established 65536 bind 65536) [ 0.601409] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.602972] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.604713] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.606548] NET: Registered protocol family 1 [ 0.608840] RPC: Registered named UNIX socket transport module. [ 0.610522] RPC: Registered udp transport module. [ 0.611436] RPC: Registered tcp transport module. [ 0.612325] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.613579] NET: Registered protocol family 44 [ 0.614800] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.616099] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.617319] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.619070] PCI: CLS 0 bytes, default 64 [ 0.620214] Unpacking initramfs... [ 2.062121] debug: unmapping init [mem 0xffff9809fcc64000-0xffff9809fffcffff] [ 2.065890] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.067591] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.069868] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.686777] Initialise system trusted keyrings [ 2.688434] Key type blacklist registered [ 2.691043] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.700527] zbud: loaded [ 2.703177] *** VALIDATE nfs *** [ 2.704157] *** VALIDATE nfs4 *** [ 2.705389] pstore: using deflate compression [ 2.713489] Platform Keyring initialized [ 2.825396] NET: Registered protocol family 38 [ 2.826701] Key type asymmetric registered [ 2.828447] Asymmetric key parser 'x509' registered [ 2.829685] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.831898] io scheduler mq-deadline registered [ 2.833283] io scheduler kyber registered [ 2.834695] io scheduler bfq registered [ 2.836886] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.839303] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.841401] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.843557] ACPI: Power Button [PWRF] [ 2.848714] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.854225] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.870076] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.896585] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.927018] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.935340] Non-volatile memory driver v1.3 [ 2.937098] Linux agpgart interface v0.103 [ 2.967882] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 2.970966] vda: detected capacity change from 0 to 75096064 [ 3.002044] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.004380] vdb: detected capacity change from 0 to 1073741824 [ 3.016034] libphy: Fixed MDIO Bus: probed [ 3.032822] usbcore: registered new interface driver usbserial_generic [ 3.034874] usbserial: USB Serial support registered for generic [ 3.036774] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.040292] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.044184] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.047892] mousedev: PS/2 mouse device common for all mice [ 3.050326] rtc_cmos 00:05: RTC can wake from S4 [ 3.054186] rtc_cmos 00:05: registered as rtc0 [ 3.057454] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.059597] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.061923] intel_pstate: CPU model not supported [ 3.069664] hid: raw HID events driver (C) Jiri Kosina [ 3.072227] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.074115] usbcore: registered new interface driver usbhid [ 3.081201] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.082733] usbhid: USB HID core driver [ 3.082853] drop_monitor: Initializing network drop monitor service [ 3.090235] Initializing XFRM netlink socket [ 3.091971] NET: Registered protocol family 10 [ 3.095736] Segment Routing with IPv6 [ 3.096779] NET: Registered protocol family 17 [ 3.098964] mpls_gso: MPLS GSO support [ 3.103812] RAS: Correctable Errors collector initialized. [ 3.106237] AVX version of gcm_enc/dec engaged. [ 3.108211] AES CTR mode by8 optimization enabled [ 3.177121] sched_clock: Marking stable (3177104890, 0)->(3892280843, -715175953) [ 3.180985] registered taskstats version 1 [ 3.183143] Loading compiled-in X.509 certificates [ 3.185780] zswap: loaded using pool lzo/zbud [ 3.208832] Key type big_key registered [ 3.221421] Key type encrypted registered [ 3.223391] ima: No TPM chip found, activating TPM-bypass! [ 3.225750] ima: Allocated hash algorithm: sha1 [ 3.227521] ima: No architecture policies found [ 3.229246] evm: Initialising EVM extended attributes: [ 3.231164] evm: security.selinux [ 3.232350] evm: security.ima [ 3.233413] evm: security.capability [ 3.234723] evm: HMAC attrs: 0x1 [ 3.236919] rtc_cmos 00:05: setting system clock to 2026-09-08 18:06:21 UTC (1788890781) [ 3.244166] debug: unmapping init [mem 0xffffffff9a203000-0xffffffff9a3fffff] [ 3.247338] debug: unmapping init [mem 0xffffffff98f82000-0xffffffff99258fff] [ 3.257038] Write protecting the kernel read-only data: 28672k [ 3.261448] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.264137] debug: unmapping init [mem 0xffffffff97f14000-0xffffffff97ffffff] [ 3.296350] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.305222] systemd[1]: Detected virtualization kvm. [ 3.307335] systemd[1]: Detected architecture x86-64. [ 3.309429] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.338960] systemd[1]: No hostname configured. [ 3.342622] systemd[1]: Set hostname to . [ 3.344591] random: systemd: uninitialized urandom read (16 bytes read) [ 3.346993] systemd[1]: Initializing machine ID from random generator. [ 3.546338] random: systemd: uninitialized urandom read (16 bytes read) [ 3.548886] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.560106] random: systemd: uninitialized urandom read (16 bytes read) [ 3.562712] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.568300] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.251418] device-mapper: uevent: version 1.0.3 [ 4.253475] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.946148] random: fast init done Starting dracut initqueue hook... [ 5.052057] virtio_net virtio0 ens2: renamed from eth0 [ 5.118016] scsi host0: ata_piix [ 5.246040] scsi host1: ata_piix [ 5.247327] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.249316] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.543859] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.746241] random: crng init done [ 9.747404] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.887535] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.262034] printk: systemd: 21 output lines suppressed due to ratelimiting [ 12.547152] SELinux: Disabled at runtime. [ 12.605793] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.611308] systemd[1]: Detected virtualization kvm. [ 12.613236] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.340956] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.345384] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.356778] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.361697] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.367117] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.376778] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.385937] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ 13.506485] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.947746] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.377290] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.384119] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.587310] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.631964] EDAC sbridge: Ver: 1.1.2 [ 16.285143] Key type dns_resolver registered [ 16.631905] NFS: Registering the id_resolver key type [ 16.633443] Key type id_resolver registered [ 16.634617] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Login Service. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. Starting Authorization Manager... [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Started System Logging Service. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-client login: [ 40.033888] libcfs: loading out-of-tree module taints kernel. [ 40.076856] Key type ._llcrypt registered [ 40.077937] Key type .llcrypt registered [ 40.224523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.228994] alg: No test for adler32 (adler32-zlib) [ 41.178541] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 41.458427] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 43.063218] Key type lgssc registered [ 43.652549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.293973] Lustre: Mounted lustre-client - version 2.17.57_103_ga91bf53 [ 90.733701] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 103.668083] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing check_logdir /tmp/testlogs/ [ 106.464498] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing yml_node [ 109.094862] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 110.797563] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 112.635460] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 113.527400] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 14:08:10 EDT 2026 [ 114.143251] Lustre: lustre-OST0000-osc-ffff980a47c94800: disconnect after 24s idle [ 123.835985] Lustre: Unmounted lustre-client [ 158.551052] Key type lgssc unregistered [ 158.728964] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 158.735975] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 158.750240] LNet: Removed LNI 192.168.201.6@tcp [ 159.562135] Key type .llcrypt unregistered [ 159.566304] Key type ._llcrypt unregistered [ 179.559616] Key type ._llcrypt registered [ 179.564783] Key type .llcrypt registered [ 179.712252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 179.723344] alg: No test for adler32 (adler32-zlib) [ 180.553423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 180.580868] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 180.591581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 182.983673] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 183.136106] LNet: 9289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 183.139241] LNetError: 9289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 183.145808] LNet: Removed LNI 192.168.201.6@tcp [ 183.587441] Key type .llcrypt unregistered [ 183.589207] Key type ._llcrypt unregistered [ 187.336549] Key type ._llcrypt registered [ 187.340359] Key type .llcrypt registered [ 188.264532] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:09:25 (1788890965) [ 188.570362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 188.591569] alg: No test for adler32 (adler32-zlib) [ 189.405159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.441677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.453340] LNet: 9696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 189.465778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.499864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 189.509475] LNet: 9703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 193.795091] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:09:31 (1788890971) [ 193.838143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.847955] LNet: 10274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.862903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.911310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 197.777990] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:09:35 (1788890975) [ 197.832330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 197.851267] LNet: 10857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 197.872829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 197.952695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 202.072248] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:09:39 (1788890979) [ 202.130368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.144454] LNet: 11441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 202.161086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.222724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 206.562408] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:09:43 (1788890983) [ 206.600203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.617643] LNet: 12026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 206.632120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.687476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 206.925601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 210.639413] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:09:48 (1788890988) [ 210.710948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.735367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.778932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 210.972376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 211.164458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 214.660682] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:09:52 (1788890992) [ 214.785586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 214.800466] LNet: 13218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 214.802902] LNet: 13218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 214.811652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.858764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 218.167866] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:09:55 (1788890995) [ 218.271721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.292136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.317658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 218.334346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 218.555895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 218.589484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 218.813812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 218.844085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 219.055639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 219.078753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 219.299968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 219.326708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 219.489778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 219.503452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 219.716948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 219.739614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 219.935800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 219.957217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 220.174448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 220.192979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 220.399966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 220.417801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 220.612161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 220.634489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 220.883737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 220.915131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 221.120627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 221.149496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 224.830193] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:10:02 (1788891002) [ 224.880484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.913431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.990271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 225.020856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 228.919400] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:10:06 (1788891006) [ 229.065452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.094585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.149416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 229.179582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 232.853555] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:10:10 (1788891010) [ 232.997345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.010199] LNet: 15653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.015948] LNet: 15653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 233.027512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.065254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 233.091409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 237.087196] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:10:14 (1788891014) [ 237.125526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.149402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 237.209277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 237.229530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 240.797624] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:10:18 (1788891018) [ 240.942394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 240.974061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.037946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 241.062327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 245.035406] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:10:22 (1788891022) [ 245.088729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.111502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 245.161563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 245.184204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 248.181978] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:10:25 (1788891025) [ 248.269830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 248.294913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.339194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 248.557678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 251.780758] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:10:29 (1788891029) [ 251.953436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 251.975152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.099930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 252.236835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 255.950715] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:10:33 (1788891033) [ 256.020341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.041930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.111666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 256.215190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 259.854257] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:10:37 (1788891037) [ 259.908608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.944565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.349236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 260.512256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 264.091446] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:10:41 (1788891041) [ 264.257823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.280550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.321656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 264.346277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 268.116935] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:10:45 (1788891045) [ 268.197319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.208501] LNet: 21065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.214771] LNet: 21065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 268.228337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.275864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.308846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 272.492992] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:10:49 (1788891049) [ 272.563480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 272.586355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 272.639851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 276.853793] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:10:54 (1788891054) [ 277.004928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.062862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 277.113342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 280.936133] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:10:58 (1788891058) [ 281.090777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.108935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.161994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 281.364884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 285.483201] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:11:02 (1788891062) [ 285.547566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.580973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.675232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 285.699527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 289.410575] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:11:06 (1788891066) [ 289.546978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.580364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.627488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 289.648879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 293.286775] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:11:10 (1788891070) [ 293.384952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 293.417257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 293.467593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 293.497405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 297.360753] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:11:14 (1788891074) [ 297.481246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.509610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.553081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 297.581186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 301.726709] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:11:19 (1788891079) [ 301.796522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.830063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.875184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 301.922890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 301.935562] LNetError: 25788:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 301.955247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 301.968657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 305.568315] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:11:22 (1788891082) [ 305.622711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.646860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.705065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ViolTPSwSB [ 305.763349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ViolTPSwSB [ 305.801648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ViolTPSwSB [ 309.423106] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:11:26 (1788891086) [ 309.584598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.614431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.651903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 309.674403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 309.712343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 313.326844] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 14:11:30 (1788891090) [ 313.445895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.472384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.529525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 313.556079] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 314.356793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 314.977888] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 316.736044] Key type lgssc registered [ 317.293947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 319.169873] LNetError: 27548:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.6@tcp99 has no available nets [ 320.779874] Key type lgssc unregistered [ 321.012708] LNetError: 28653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 322.092908] LNet: Removed LNI 192.168.201.6@tcp [ 322.444409] Key type .llcrypt unregistered [ 322.449841] Key type ._llcrypt unregistered [ 325.472114] Key type ._llcrypt registered [ 325.473726] Key type .llcrypt registered [ 326.196730] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:11:43 (1788891103) [ 326.462393] Key type .llcrypt unregistered [ 326.464784] Key type ._llcrypt unregistered [ 329.662916] Key type ._llcrypt registered [ 329.664386] Key type .llcrypt registered [ 329.758836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 329.769524] alg: No test for adler32 (adler32-zlib) [ 330.585916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.627452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 330.640150] LNet: 29562:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 330.644658] LNet: 29562:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 330.648735] LNet: 29562:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 330.658660] LNet: Message forwarding will be enabled in 2s [ 333.727217] LNet: Message forwarding enabled [ 333.903967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 333.913793] LNet: Message forwarding will stop in 2s to 6s [ 337.831445] LNet: Message forwarding disabled [ 338.220098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 338.234593] LNet: 29661:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 338.239918] LNet: 29661:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 338.244605] LNet: 29661:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 338.254209] LNet: Message forwarding will be enabled in 2s [ 338.335086] LNet: 29674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.724268] Key type .llcrypt unregistered [ 338.725700] Key type ._llcrypt unregistered [ 345.956633] Key type ._llcrypt registered [ 345.960378] Key type .llcrypt registered [ 346.808438] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:12:04 (1788891124) [ 347.005294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 347.013530] alg: No test for adler32 (adler32-zlib) [ 347.892652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.931381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 352.161957] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:12:09 (1788891129) [ 352.221724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.236549] LNet: 31067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.255274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.300597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 352.337333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 352.363956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 352.386979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 352.409887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 352.431425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 352.462302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 352.496373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 352.521940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 352.544909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 352.562660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 352.584319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 352.613231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 352.642402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 352.674183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 352.699282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 352.726359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 352.757386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 352.784324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 352.800896] LNetError: 31119:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 356.778322] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:12:14 (1788891134) [ 356.856761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.874483] LNet: 31685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.889645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.942543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 360.858255] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:12:18 (1788891138) [ 361.898945] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg106-server [ 362.974263] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:12:20 (1788891140) [ 363.017053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.043071] LNet: 32614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.062863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.122606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 363.141736] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 363.625643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[7-11]@tcp [ 363.645101] LNet: 32755:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 363.664483] LNetError: 32631:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.7@tcp has gone from up to down [ 363.685914] LNetError: 32626:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.10@tcp is being used as a gateway but routing feature is not turned on [ 366.752834] LNetError: 32630:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.9@tcp has gone from up to down [ 366.760908] LNetError: 32630:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 367.879257] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:12:25 (1788891145) [ 368.828423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg106-server [ 369.824898] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:12:27 (1788891147) [ 369.931539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.954821] LNet: 33690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.962199] LNetError: 33690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 369.976942] LNet: Removed LNI 192.168.201.6@tcp [ 369.991643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.026254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 370.041612] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 370.485643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[7-12/2]@tcp [ 370.494772] LNet: 33807:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 370.499475] LNet: 33807:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 370.510970] LNetError: 33698:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.7@tcp has gone from up to down [ 370.512778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[7-12/2]@tcp [ 370.516252] LNetError: 33698:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 370.525947] LNetError: 33694:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.11@tcp is being used as a gateway but routing feature is not turned on [ 370.530576] LNetError: 33694:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 374.297812] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:12:31 (1788891151) [ 374.430122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.444531] LNet: 34398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.448883] LNetError: 34398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 374.466641] LNet: Removed LNI 192.168.201.6@tcp [ 374.498275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.583622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 374.630683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.654428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.699611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 374.729809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.753487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.778378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 374.821543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.847189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.886918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 374.947340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.971633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.999287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 375.055996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.097909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.154650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.192803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.219347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.254632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 375.315810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.345916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.398187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.464227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.494560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.522828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 375.574467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.592980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.629312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.672272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.684983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.712976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 375.751802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.777959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.825354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.791588] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:12:37 (1788891157) [ 381.092790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg106-server [ 382.133633] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:12:39 (1788891159) [ 383.539473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg106-server [ 384.902661] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:12:42 (1788891162) [ 384.966972] LNet: 35790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 384.973698] LNet: 35790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 385.456792] Key type .llcrypt unregistered [ 385.458515] Key type ._llcrypt unregistered [ 389.262263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 389.353149] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 389.357259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 389.363357] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 391.742697] Key type ._llcrypt registered [ 391.744179] Key type .llcrypt registered [ 391.876190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 391.885342] alg: No test for adler32 (adler32-zlib) [ 392.761541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.787712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 392.803073] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 393.202561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 393.216057] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 394.400486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 394.417712] LNet: Removed LNI 10.1.2.3@tcp [ 394.516176] LNet: 36516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.520687] LNetError: 36516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 394.535868] LNet: Removed LNI 192.168.201.6@tcp [ 394.919809] Key type .llcrypt unregistered [ 394.923357] Key type ._llcrypt unregistered [ 401.582148] Key type ._llcrypt registered [ 401.586094] Key type .llcrypt registered [ 402.471872] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:12:59 (1788891179) [ 402.818417] Key type .llcrypt unregistered [ 402.819806] Key type ._llcrypt unregistered [ 406.719152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 406.828240] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 406.839574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 406.842737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 409.191146] Key type ._llcrypt registered [ 409.192383] Key type .llcrypt registered [ 409.315263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 409.325244] alg: No test for adler32 (adler32-zlib) [ 410.183711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.222318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 410.236446] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 410.728654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 410.741681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 412.494418] LNet: 38005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.503059] LNetError: 38005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 412.513163] LNet: Removed LNI 10.1.2.3@tcp [ 412.525825] LNet: Removed LNI 192.168.201.6@tcp [ 412.927088] Key type .llcrypt unregistered [ 412.930311] Key type ._llcrypt unregistered [ 419.330487] Key type ._llcrypt registered [ 419.332834] Key type .llcrypt registered [ 420.134257] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:13:17 (1788891197) [ 420.453384] Key type .llcrypt unregistered [ 420.454718] Key type ._llcrypt unregistered [ 424.038093] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 424.154789] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 424.157241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 424.161289] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 426.485152] Key type ._llcrypt registered [ 426.487018] Key type .llcrypt registered [ 426.632418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 426.645918] alg: No test for adler32 (adler32-zlib) [ 427.490825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.523762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 427.537499] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 428.656127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 428.668210] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 429.741273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 429.749643] LNet: Removed LNI 10.1.2.3@tcp [ 429.756981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 429.769620] LNet: Removed LNI 10.1.2.31@tcp [ 429.776535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 429.788643] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 430.136097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 430.152917] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 430.658432] LNet: 39917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.661925] LNetError: 39917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 430.667929] LNet: Removed LNI 10.1.2.31@tcp [ 430.676652] LNet: Removed LNI 10.1.2.3@tcp [ 431.069899] Key type .llcrypt unregistered [ 431.070983] Key type ._llcrypt unregistered [ 436.846191] Key type ._llcrypt registered [ 436.847641] Key type .llcrypt registered [ 437.656112] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:13:35 (1788891215) [ 437.901507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 437.911353] alg: No test for adler32 (adler32-zlib) [ 438.730761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.763563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 438.782100] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 439.198735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.213353] LNet: 40865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 439.217832] LNetError: 40865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 439.227765] LNet: Removed LNI 192.168.201.6@tcp [ 439.240982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.268722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.6@tcp [ 439.283633] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 439.733528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.6@tcp [ 439.758062] LNet: Removed LNI 192.168.201.6@tcp [ 439.802916] LNet: 41000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.245706] Key type .llcrypt unregistered [ 440.248518] Key type ._llcrypt unregistered [ 446.528693] Key type ._llcrypt registered [ 446.529894] Key type .llcrypt registered [ 447.219554] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:13:44 (1788891224) [ 448.161042] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg106-server [ 449.058411] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:13:46 (1788891226) [ 449.379494] Key type .llcrypt unregistered [ 449.381073] Key type ._llcrypt unregistered [ 452.970523] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 453.098874] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 453.106163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 453.109422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 455.374893] Key type ._llcrypt registered [ 455.375983] Key type .llcrypt registered [ 455.454768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 455.460231] alg: No test for adler32 (adler32-zlib) [ 456.272390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.307131] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 456.606532] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 457.831081] LNet: 42827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 457.834299] LNetError: 42827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 457.838053] LNet: Removed LNI 192.168.201.6@tcp1 [ 457.852489] LNet: Removed LNI 10.1.2.3@tcp2 [ 458.158773] Key type .llcrypt unregistered [ 458.159889] Key type ._llcrypt unregistered [ 464.853843] Key type ._llcrypt registered [ 464.856669] Key type .llcrypt registered [ 465.785365] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:14:02 (1788891242) [ 466.075194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 466.085388] alg: No test for adler32 (adler32-zlib) [ 466.943207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.971244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 466.989168] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 467.505104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.522211] LNet: 43793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.526885] LNetError: 43793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.542306] LNet: Removed LNI 192.168.201.6@tcp [ 467.566247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.617625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 467.636235] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 471.870499] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 14:14:09 (1788891249) [ 472.027768] LNet: 44384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.032878] LNetError: 44384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 472.049885] LNet: Removed LNI 192.168.201.6@tcp [ 472.404537] Key type .llcrypt unregistered [ 472.408402] Key type ._llcrypt unregistered [ 476.610489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 476.790466] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 476.796258] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 476.798529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 479.173339] Key type ._llcrypt registered [ 479.175591] Key type .llcrypt registered [ 479.292450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 479.303884] alg: No test for adler32 (adler32-zlib) [ 480.123606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.153989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 480.167647] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 480.627089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 480.638984] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 481.888924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.900406] LNet: 45121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.904437] LNetError: 45121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 481.907597] LNet: Removed LNI 10.1.2.3@tcp [ 481.914896] LNet: Removed LNI 192.168.201.6@tcp [ 481.928530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.963047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 481.983154] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 481.986108] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 482.219781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.237692] LNet: Removed LNI 10.1.2.3@tcp [ 482.250203] LNet: Removed LNI 192.168.201.6@tcp [ 482.264786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.297838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 482.316602] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 482.319742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 482.541392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 482.555493] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 483.183906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.200901] LNet: 45310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.203126] LNet: 45310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 483.206910] LNetError: 45310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 483.210891] LNetError: 45310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 483.214307] LNet: Removed LNI 10.1.2.3@tcp [ 483.215964] LNet: Removed LNI 192.168.201.6@tcp [ 483.227785] LNet: Removed LNI 192.168.201.6@tcp2 [ 483.242349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.271123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 483.284114] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 483.288496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 483.292928] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 487.378721] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 14:14:24 (1788891264) [ 487.468795] LNet: 45901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 487.474510] LNetError: 45901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 487.479299] LNetError: 45901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 487.490247] LNet: Removed LNI 10.1.2.3@tcp [ 487.493498] LNet: Removed LNI 192.168.201.6@tcp [ 487.505973] LNet: Removed LNI 192.168.201.6@tcp2 [ 487.888965] Key type .llcrypt unregistered [ 487.891870] Key type ._llcrypt unregistered [ 491.316275] Key type ._llcrypt registered [ 491.317404] Key type .llcrypt registered [ 491.425340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 491.433961] alg: No test for adler32 (adler32-zlib) [ 492.327158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.369630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 492.391447] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 492.836695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.854906] LNet: 46298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.859223] LNetError: 46298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.873688] LNet: Removed LNI 192.168.201.6@tcp [ 492.889693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.938791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 492.956383] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 493.229441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.268076] LNet: Removed LNI 192.168.201.6@tcp [ 493.281433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.338823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 493.353235] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 493.655767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.667306] LNet: 46358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 493.669881] LNet: 46358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 493.673823] LNetError: 46358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 493.677234] LNetError: 46358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 493.692228] LNet: Removed LNI 192.168.201.6@tcp [ 493.710601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.750361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 493.762244] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 494.006891] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 494.059472] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 494.061816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 494.064831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 496.322872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.332576] LNet: 46438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 496.335195] LNetError: 46438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 496.348519] LNet: Removed LNI 192.168.201.6@tcp [ 496.357465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.415346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 496.435812] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 500.251399] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 14:14:37 (1788891277) [ 500.344784] LNet: 47034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.347550] LNetError: 47034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.359388] LNet: Removed LNI 192.168.201.6@tcp [ 500.716838] Key type .llcrypt unregistered [ 500.718521] Key type ._llcrypt unregistered [ 504.274254] Key type ._llcrypt registered [ 504.283092] Key type .llcrypt registered [ 504.463485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 504.470690] alg: No test for adler32 (adler32-zlib) [ 505.329853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.361723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 505.374813] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 505.842383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.853317] LNet: 47430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 505.861344] LNetError: 47430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 505.870772] LNet: Removed LNI 192.168.201.6@tcp [ 505.879414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.905640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 505.916141] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 509.585135] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 14:14:47 (1788891287) [ 509.700869] LNet: 48022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.703290] LNetError: 48022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 509.710225] LNet: Removed LNI 192.168.201.6@tcp [ 510.104434] Key type .llcrypt unregistered [ 510.106128] Key type ._llcrypt unregistered [ 513.647649] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 513.798648] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 513.803720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 513.806890] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 516.137803] Key type ._llcrypt registered [ 516.139264] Key type .llcrypt registered [ 516.270691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 516.289497] alg: No test for adler32 (adler32-zlib) [ 517.132589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.176471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 517.191689] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 517.685336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.707132] LNet: 48469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.714637] LNetError: 48469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.728067] LNet: Removed LNI 192.168.201.6@tcp [ 517.746064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.810191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 517.835669] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 522.093786] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 14:14:59 (1788891299) [ 522.263903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.278916] LNet: 49067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.283777] LNetError: 49067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 522.292324] LNet: Removed LNI 192.168.201.6@tcp [ 522.304556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.383129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 526.162194] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 14:15:03 (1788891303) [ 526.260592] LNet: 49645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 526.680391] Key type .llcrypt unregistered [ 526.681569] Key type ._llcrypt unregistered [ 530.491961] Key type ._llcrypt registered [ 530.494530] Key type .llcrypt registered [ 530.636381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 530.647661] alg: No test for adler32 (adler32-zlib) [ 531.484544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.517149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 531.531705] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 532.060776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.082651] LNet: 50042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 532.088501] LNetError: 50042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.104897] LNet: Removed LNI 192.168.201.6@tcp [ 532.127823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.179420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 532.194169] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 536.308734] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 14:15:13 (1788891313) [ 536.349325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.366251] LNet: 50628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.375708] LNetError: 50628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 536.392034] LNet: Removed LNI 192.168.201.6@tcp [ 536.413565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.462884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 540.398483] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 14:15:17 (1788891317) [ 541.378962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 542.326310] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 14:15:19 (1788891319) [ 543.236183] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 544.321728] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 14:15:21 (1788891321) [ 545.236222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 546.165632] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 14:15:23 (1788891323) [ 547.074979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 548.135927] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 14:15:25 (1788891325) [ 549.093385] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 550.150627] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 14:15:27 (1788891327) [ 550.227910] LNet: 52676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 550.676627] Key type .llcrypt unregistered [ 550.678440] Key type ._llcrypt unregistered [ 554.331915] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 554.384839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 554.390567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 554.392677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 556.783956] Key type ._llcrypt registered [ 556.785711] Key type .llcrypt registered [ 556.899868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 556.907978] alg: No test for adler32 (adler32-zlib) [ 557.761358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.808991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 557.823488] LNet: Added LNI 192.168.201.6@tcp [8/512/0/180] [ 557.831940] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 557.834223] LNet: 53001:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 557.919473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.934781] LNet: 53020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.940802] LNetError: 53020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 557.946935] LNet: Removed LNI 10.1.2.3@tcp [ 557.950829] LNetError: 53020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.967730] LNet: Removed LNI 192.168.201.6@tcp [ 557.986819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.032928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 558.049466] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 558.095980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.108740] LNet: 53044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.112737] LNetError: 53044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.126861] LNet: Removed LNI 192.168.201.6@tcp [ 558.149041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.208610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 558.223067] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 558.226039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 558.227918] LNet: 53052:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 558.272278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.283243] LNet: 53070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.286294] LNetError: 53070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 558.290755] LNet: Removed LNI 10.1.2.3@tcp [ 558.292937] LNetError: 53070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.302697] LNet: Removed LNI 192.168.201.6@tcp [ 558.316168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.358774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 558.370946] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 558.372945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 558.374224] LNet: 53078:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 558.501494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.521239] LNet: 53104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.527831] LNetError: 53104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 558.532238] LNet: Removed LNI 10.1.2.3@tcp [ 558.533797] LNetError: 53104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.556942] LNet: Removed LNI 192.168.201.6@tcp [ 558.585035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.644268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 558.658647] LNet: Added LNI 192.168.201.6@tcp [8/256/0/182] [ 558.664629] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 558.717746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.732952] LNet: 53132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 558.739576] LNetError: 53132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 558.742690] LNet: Removed LNI 10.1.2.3@tcp [ 558.744456] LNetError: 53132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 558.757050] LNet: Removed LNI 192.168.201.6@tcp [ 558.767734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.822864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 558.839288] LNet: Added LNI 192.168.201.6@tcp [8/256/0/182] [ 558.842917] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 558.845072] LNet: 53140:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 558.892489] LNet: Removed LNI 10.1.2.3@tcp [ 558.903320] LNet: Removed LNI 192.168.201.6@tcp [ 559.288564] Key type .llcrypt unregistered [ 559.292430] Key type ._llcrypt unregistered [ 565.842565] Key type ._llcrypt registered [ 565.844261] Key type .llcrypt registered [ 566.714575] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 14:15:43 (1788891343) [ 567.598417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 568.519239] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:15:46 (1788891346) [ 568.829334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 568.845631] alg: No test for adler32 (adler32-zlib) [ 569.702277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.756990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 569.777897] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 570.320574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.6 [ 570.339747] LNet: 54400:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 570.345983] LNet: Removed LNI 192.168.201.6@tcp [ 574.557098] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:15:51 (1788891351) [ 574.732231] LNet: 54970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.742857] LNetError: 54970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 574.762451] LNet: Removed LNI 192.168.201.6@tcp [ 575.233460] Key type .llcrypt unregistered [ 575.237976] Key type ._llcrypt unregistered [ 580.000319] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 580.165633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 582.667874] Key type ._llcrypt registered [ 582.670368] Key type .llcrypt registered [ 582.797581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 582.814939] alg: No test for adler32 (adler32-zlib) [ 584.058493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.240051] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:16:05 (1788891365) [ 588.415560] LNet: 55879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.419112] LNetError: 55879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 588.429171] LNet: Removed LNI 10.1.2.3@tcp [ 588.803555] Key type .llcrypt unregistered [ 588.807460] Key type ._llcrypt unregistered [ 592.702466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 592.852164] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 592.856709] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 592.859402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 595.320470] Key type ._llcrypt registered [ 595.323177] Key type .llcrypt registered [ 595.454097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.468620] alg: No test for adler32 (adler32-zlib) [ 596.352688] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 600.460235] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:16:17 (1788891377) [ 601.383459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 602.360378] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:16:19 (1788891379) [ 602.441109] LNet: 57055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.444958] LNetError: 57055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 602.458540] LNet: Removed LNI 10.1.2.3@tcp [ 602.809492] Key type .llcrypt unregistered [ 602.810727] Key type ._llcrypt unregistered [ 606.051651] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 606.173350] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 606.178145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 606.183202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 608.582173] Key type ._llcrypt registered [ 608.584152] Key type .llcrypt registered [ 608.679819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.686267] alg: No test for adler32 (adler32-zlib) [ 609.531621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.587352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.250139] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:16:31 (1788891391) [ 614.510833] LNet: 58284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.514841] LNetError: 58284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 614.528928] LNet: Removed LNI 10.1.2.3@tcp [ 614.848562] Key type .llcrypt unregistered [ 614.849895] Key type ._llcrypt unregistered [ 617.917098] Key type ._llcrypt registered [ 617.918690] Key type .llcrypt registered [ 618.035106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 618.041804] alg: No test for adler32 (adler32-zlib) [ 618.901495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 618.915061] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 624.079467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 624.497218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_interrupt -r 1 [ 624.523173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_interrupt -r 1 [ 624.543763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 624.553121] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 624.669876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 624.696213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 624.709805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 624.727135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 624.749754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 624.770650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 624.791234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 624.809914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 624.821077] LNet: 58718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 624.836149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 624.852185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 624.865190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 624.880370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 624.901674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 624.931337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 624.957714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 625.288815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_dropped -r 1 [ 625.312127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_dropped -r 1 [ 625.331883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 625.341165] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 625.469237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 625.490919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 625.513764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 625.537842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 625.563993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 625.586446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 625.607340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 625.627978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 625.641269] LNet: 58827:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 625.656617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 625.674738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 625.699217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 625.723453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 625.745283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 625.773559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 625.797832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 626.138449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_aborted -r 1 [ 626.170577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_aborted -r 1 [ 626.195205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 626.362172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 626.383141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 626.413751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 626.441448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 626.466168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 626.493043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 626.514754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 626.536403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 626.558796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 626.589899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 626.610269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 626.633514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 626.658715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 626.681539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 626.708631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 627.032527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_no_route -r 1 [ 627.056412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_no_route -r 1 [ 627.092392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 627.112989] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 627.118236] LNet: Skipped 1 previous similar message [ 627.265526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 627.291508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 627.314758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 627.335366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 627.359915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 627.379607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 627.400609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 627.420869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 627.431530] LNet: 59045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 627.437973] LNet: 59045:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 627.450554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 627.468600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 627.490203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 627.506244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 627.522951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 627.539396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 627.555569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 627.893900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_timeout -r 1 [ 627.919729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_timeout -r 1 [ 627.940133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 628.094335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 628.121351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 628.127219] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 628.145554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 628.166669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 628.192342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 628.210628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 628.232232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 628.245936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 628.266222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 628.293384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 628.324472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 628.342583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 628.361925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 628.379636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 628.401788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 628.779955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_error -r 1 [ 628.816765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_error -r 1 [ 628.849507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 628.990523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 629.007569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 629.027403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 629.047986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 629.068395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 629.085725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 629.100454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 629.123164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 629.146494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 629.165729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 629.184180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 629.204895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 629.226755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 629.250101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 629.269113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 629.429469] LNet: 59305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.434263] LNetError: 59305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.455689] LNet: Removed LNI 192.168.201.6@tcp [ 629.725695] Key type .llcrypt unregistered [ 629.727561] Key type ._llcrypt unregistered [ 638.877226] Key type ._llcrypt registered [ 638.879517] Key type .llcrypt registered [ 639.690443] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:16:57 (1788891417) [ 640.246228] Key type .llcrypt unregistered [ 640.248030] Key type ._llcrypt unregistered [ 643.526662] Key type ._llcrypt registered [ 643.528316] Key type .llcrypt registered [ 643.644815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.651948] alg: No test for adler32 (adler32-zlib) [ 644.519232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 644.536354] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 648.902349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 654.513393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.532712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 654.546819] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 655.205123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_interrupt -r 1 [ 655.222868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_interrupt -r 1 [ 655.246908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_interrupt -r 1 [ 655.272304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_interrupt -r 1 [ 655.293304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_interrupt -r 1 [ 655.316549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_interrupt -r 1 [ 655.340675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_interrupt -r 1 [ 655.364060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_interrupt -r 1 [ 655.384831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 655.397931] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 655.401478] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 655.508556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 655.525293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 655.544076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 655.562792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 655.580953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 655.595449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 655.610468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 655.625936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 655.633874] LNet: 60764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 655.644645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 655.664564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 655.686538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 655.702900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 655.722507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 655.742176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 655.760385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 656.018283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_dropped -r 1 [ 656.042041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_dropped -r 1 [ 656.062863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_dropped -r 1 [ 656.086919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_dropped -r 1 [ 656.105588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_dropped -r 1 [ 656.133501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_dropped -r 1 [ 656.158799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_dropped -r 1 [ 656.185921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_dropped -r 1 [ 656.204450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 656.213832] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 656.217312] LNet: Skipped 2 previous similar messages [ 656.325715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 656.342890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 656.364859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 656.381560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 656.397840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 656.418269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 656.436082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 656.454134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 656.463050] LNet: 60885:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 656.474724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 656.493291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 656.517260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 656.536461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 656.556714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 656.586818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 656.611353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 656.909238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_aborted -r 1 [ 656.937678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_aborted -r 1 [ 656.971167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_aborted -r 1 [ 656.997393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_aborted -r 1 [ 657.022237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_aborted -r 1 [ 657.054067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_aborted -r 1 [ 657.075378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_aborted -r 1 [ 657.102264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_aborted -r 1 [ 657.125396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 657.290593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 657.311761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 657.328213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 657.357866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 657.383688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 657.412323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 657.430856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 657.450994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 657.472521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 657.493268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 657.515924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 657.534992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 657.558773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 657.582849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 657.611338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 657.895800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_no_route -r 1 [ 657.924219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_no_route -r 1 [ 657.947259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_no_route -r 1 [ 657.978221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_no_route -r 1 [ 657.999763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_no_route -r 1 [ 658.024804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_no_route -r 1 [ 658.050868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_no_route -r 1 [ 658.078225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_no_route -r 1 [ 658.108062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 658.118572] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 658.121920] LNet: Skipped 7 previous similar messages [ 658.237335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 658.251756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 658.269193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 658.287719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 658.307792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 658.325633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 658.346967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 658.367413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 658.378960] LNet: 61127:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 658.382060] LNet: 61127:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 658.391853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 658.411292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 658.431489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 658.454157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 658.475054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 658.494475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 658.522589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 658.837495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_timeout -r 1 [ 658.860872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_timeout -r 1 [ 658.883488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_timeout -r 1 [ 658.904187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_timeout -r 1 [ 658.924845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_timeout -r 1 [ 658.948146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_timeout -r 1 [ 658.977422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_timeout -r 1 [ 658.997377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_timeout -r 1 [ 659.026420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 659.198494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 659.222265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 659.254188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 659.275732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 659.293272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 659.314531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 659.334542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 659.359616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 659.386902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 659.411935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 659.444429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 659.469832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 659.492705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 659.523334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 659.545767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 659.861019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_error -r 1 [ 659.879940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_error -r 1 [ 659.898391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_error -r 1 [ 659.928617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_error -r 1 [ 659.952658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_error -r 1 [ 659.979454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_error -r 1 [ 659.999944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_error -r 1 [ 660.017824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_error -r 1 [ 660.041866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 660.180672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 660.195553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 660.227940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 660.253835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 660.277829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 660.309223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 660.336793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 660.356268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 660.392988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 660.425979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 660.451506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 660.477756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 660.498979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 660.524712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 660.545769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 663.676677] LNet: 61445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.680610] LNetError: 61445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.684594] LNet: Removed LNI 192.168.201.6@tcp [ 664.741849] LNet: Removed LNI 192.168.201.6@tcp1 [ 664.943574] Key type .llcrypt unregistered [ 664.944734] Key type ._llcrypt unregistered [ 673.246185] Key type ._llcrypt registered [ 673.247613] Key type .llcrypt registered [ 673.845354] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:17:31 (1788891451) [ 674.288466] Key type .llcrypt unregistered [ 674.289860] Key type ._llcrypt unregistered [ 677.019576] Key type ._llcrypt registered [ 677.021522] Key type .llcrypt registered [ 677.097478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 677.103345] alg: No test for adler32 (adler32-zlib) [ 677.943129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 677.961942] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 682.234409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 682.755789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_dropped -r 1 [ 682.787813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_dropped -r 1 [ 682.834362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 682.849412] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 683.026240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 683.051451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 683.073472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 683.097790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 683.115208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 683.140535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 683.169628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 683.197316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 683.211477] LNet: 62692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 683.226249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 683.261435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 683.278742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 683.303996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 683.328936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 683.344198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 683.365258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 683.643713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_error -r 1 [ 683.670410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_error -r 1 [ 683.695985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 683.707551] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 683.828994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 683.843323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 683.861910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 683.880499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 683.897661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 683.912398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 683.928828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 683.946182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 683.957561] LNet: 62801:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 683.970681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 683.994973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 684.015501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 684.041968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 684.069652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 684.084346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 684.103536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 684.411452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_timeout -r 1 [ 684.439142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_timeout -r 1 [ 684.464604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 684.607448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 684.627656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 684.644757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 684.658114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 684.680243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 684.705648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 684.722630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 684.743443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 684.768594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 684.782538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 684.797817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 684.813671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 684.827851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 684.843762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 684.866633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 685.027217] LNet: 62951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.032429] LNetError: 62951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.064124] LNet: Removed LNI 192.168.201.6@tcp [ 686.328525] Key type .llcrypt unregistered [ 686.331398] Key type ._llcrypt unregistered [ 696.964420] Key type ._llcrypt registered [ 696.966551] Key type .llcrypt registered [ 697.764543] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:17:55 (1788891475) [ 698.445379] Key type .llcrypt unregistered [ 698.447586] Key type ._llcrypt unregistered [ 702.205798] Key type ._llcrypt registered [ 702.207842] Key type .llcrypt registered [ 702.325143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 702.331961] alg: No test for adler32 (adler32-zlib) [ 703.177385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 703.193444] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 707.692261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 712.385313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.432932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 712.449478] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 713.252411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_dropped -r 1 [ 713.295671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e remote_dropped -r 1 [ 713.324402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_dropped -r 1 [ 713.344967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e remote_dropped -r 1 [ 713.376526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e remote_dropped -r 1 [ 713.411856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e remote_dropped -r 1 [ 713.441097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e remote_dropped -r 1 [ 713.466852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e remote_dropped -r 1 [ 713.504508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 713.524657] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 713.529596] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.106@tcp [ 713.683384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 713.706895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 713.728457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 713.745483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 713.766175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 713.787584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 713.808588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 713.832136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 713.846359] LNet: 64411:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 713.859310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 713.876809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 713.897511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 713.921094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 713.940893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 713.962827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 713.979871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 713.996253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 714.285927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_error -r 1 [ 714.317739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e remote_error -r 1 [ 714.337526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_error -r 1 [ 714.361878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e remote_error -r 1 [ 714.382229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e remote_error -r 1 [ 714.404586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e remote_error -r 1 [ 714.425699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e remote_error -r 1 [ 714.448270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e remote_error -r 1 [ 714.473792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 714.489190] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 714.497204] LNet: Skipped 2 previous similar messages [ 714.643291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 714.664657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 714.686597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 714.712737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 714.734517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 714.759820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 714.779555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 714.797951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 714.808785] LNet: 64534:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 714.818811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 714.837862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 714.860159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 714.881105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 714.919880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 714.945947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 714.972459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 714.994993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 715.254620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_timeout -r 1 [ 715.273365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e remote_timeout -r 1 [ 715.293820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_timeout -r 1 [ 715.313451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e remote_timeout -r 1 [ 715.332438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e remote_timeout -r 1 [ 715.356533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e remote_timeout -r 1 [ 715.387845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e remote_timeout -r 1 [ 715.408272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e remote_timeout -r 1 [ 715.430732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.106@tcp [ 715.555806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 715.574634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 715.597543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 715.623484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 715.642995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 715.666507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 715.687641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 715.716358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 715.747639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 715.767823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 715.795726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 715.818468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 715.840278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 715.864312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 715.888643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 715.909354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 718.946390] LNet: 64735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.953544] LNetError: 64735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.960364] LNet: Removed LNI 192.168.201.6@tcp [ 718.968853] LNet: Removed LNI 192.168.201.6@tcp1 [ 719.255768] Key type .llcrypt unregistered [ 719.257711] Key type ._llcrypt unregistered [ 728.621865] Key type ._llcrypt registered [ 728.624263] Key type .llcrypt registered [ 729.479434] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:18:26 (1788891506) [ 730.500900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 731.441527] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:18:28 (1788891508) [ 731.998589] Key type .llcrypt unregistered [ 732.000307] Key type ._llcrypt unregistered [ 735.457113] Key type ._llcrypt registered [ 735.458586] Key type .llcrypt registered [ 735.562401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.572227] alg: No test for adler32 (adler32-zlib) [ 736.438312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 736.471402] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 740.999251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 741.414723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e network_timeout -r 1 [ 741.447390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e network_timeout -r 1 [ 741.472723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 741.496412] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 741.601024] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 741.678118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 741.701721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 741.728562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 741.754185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 741.777472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 741.811186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 741.828134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 741.849474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 741.862125] LNet: 66270:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 741.876825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 741.904284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 741.929995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 741.955212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 741.981606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 742.008939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 742.191749] LNet: 66309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.198397] LNetError: 66309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.206848] LNet: Removed LNI 192.168.201.6@tcp [ 742.539472] Key type .llcrypt unregistered [ 742.544593] Key type ._llcrypt unregistered [ 752.864872] Key type ._llcrypt registered [ 752.866675] Key type .llcrypt registered [ 752.962527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 752.970977] alg: No test for adler32 (adler32-zlib) [ 753.823525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 753.840939] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 758.609833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 763.399414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.419595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 763.432702] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 764.227609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e network_timeout -r 1 [ 764.253296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e network_timeout -r 1 [ 764.275700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e network_timeout -r 1 [ 764.296295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e network_timeout -r 1 [ 764.315590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e network_timeout -r 1 [ 764.344634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e network_timeout -r 1 [ 764.363735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e network_timeout -r 1 [ 764.382871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e network_timeout -r 1 [ 764.406150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 764.416622] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 764.549471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 764.564440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 764.587660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 764.605639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 764.630597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 764.654703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 764.670285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 764.690416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 764.704484] LNet: 67181:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 764.717826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 764.740146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 764.753765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 764.776634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 764.796357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 764.816484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 767.435815] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 767.671670] LNet: 67254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.681525] LNetError: 67254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.690233] LNet: Removed LNI 192.168.201.6@tcp [ 767.700982] LNet: Removed LNI 192.168.201.6@tcp1 [ 767.981587] Key type .llcrypt unregistered [ 767.983083] Key type ._llcrypt unregistered [ 777.057779] Key type ._llcrypt registered [ 777.059585] Key type .llcrypt registered [ 777.765860] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:19:15 (1788891555) [ 777.924348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 777.935105] alg: No test for adler32 (adler32-zlib) [ 778.768734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.811692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.827951] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 779.308229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 779.319675] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 779.852199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 779.897036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 779.925220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 779.952373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 779.973894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 779.994874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 780.017869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.6@tcp [ 780.031485] LNetError: 68351:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.6@tcp to recovery [ 780.832704] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 780.834246] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 782.879723] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 783.357154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 786.976172] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 787.750236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 787.780060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 787.828880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.842984] LNet: 68468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.847958] LNetError: 68468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.896226] LNet: Removed LNI 192.168.201.6@tcp [ 788.898183] LNetError: 68468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.912372] LNet: Removed LNI 192.168.201.6@tcp1 [ 788.935301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.979410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.999364] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 789.394253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 789.418260] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 789.937412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 789.958434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 789.984406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 790.010886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 790.029715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 790.049349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 790.071615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 790.093569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.6@tcp [ 790.108144] LNetError: 68741:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.6@tcp to recovery [ 791.008146] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 793.433513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 797.151579] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 797.155288] LNet: Skipped 1 previous similar message [ 802.119849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 802.140942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 802.169099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 806.372864] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:19:43 (1788891583) [ 806.527406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.547526] LNet: 69468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 806.556506] LNetError: 69468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 806.559909] LNet: Removed LNI 192.168.201.6@tcp [ 806.570418] LNet: Removed LNI 192.168.201.6@tcp1 [ 806.585682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.633750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.651135] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 807.182692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 807.199308] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 807.852458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 807.910795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 807.937961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 807.964780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 807.995595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 808.027369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 808.072599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.6@tcp --health 0 [ 808.140183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 808.674476] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.6@tcp [ 809.696034] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 809.707908] LNet: Skipped 1 previous similar message [ 823.034370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 823.139750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 823.158382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 823.177791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.6@tcp --health 500 [ 823.238577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 829.589658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 829.628927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.643286] LNet: 69959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.647968] LNetError: 69959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.651073] LNetError: 69959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 829.654263] LNet: Removed LNI 192.168.201.6@tcp [ 829.661505] LNet: Removed LNI 192.168.201.6@tcp1 [ 829.671554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.696385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.714766] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 830.122595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 830.136599] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 830.647533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 830.677168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 830.712027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 830.732635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 830.760729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 830.788691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 830.824596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 830.868965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.6@tcp --health 0 [ 831.715778] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 831.731222] LNet: Skipped 3 previous similar messages [ 832.069698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 842.935520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 842.961776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 842.991593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 846.972623] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:20:24 (1788891624) [ 847.423210] LNet: 71015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.427651] LNetError: 71015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.432128] LNetError: 71015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 847.436808] LNet: Removed LNI 192.168.201.6@tcp [ 847.447073] LNet: Removed LNI 192.168.201.6@tcp1 [ 847.830307] Key type .llcrypt unregistered [ 847.832159] Key type ._llcrypt unregistered [ 851.771964] Key type ._llcrypt registered [ 851.773196] Key type .llcrypt registered [ 851.879526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 851.890501] alg: No test for adler32 (adler32-zlib) [ 852.795236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 852.820102] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 859.468383] Lustre: DEBUG MARKER: Initial discovery [ 859.510950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.106@tcp [ 861.011522] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 861.167793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 861.201752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.106@tcp [ 861.227424] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 861.247879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 861.278679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.106@tcp -r 1 -m GET -l 3 [ 861.315446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.106@tcp -r 1 -m GET -e local_error [ 861.345269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.106@tcp -r 1 -m PUT -l 6 [ 862.048918] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 862.296540] Lustre: DEBUG MARKER: Force oleg106-client.virtnet to discover 192.168.201.106@tcp (in background) [ 862.376627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.106@tcp [ 863.351395] Lustre: DEBUG MARKER: Force oleg106-server to discover 192.168.201.6@tcp [ 865.109590] Lustre: DEBUG MARKER: Wait for 71770 [ 865.440172] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 866.466314] Lustre: DEBUG MARKER: Finished wait on 71770 [ 866.496973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 866.521244] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 866.527068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 866.668392] LNet: 72224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.672133] LNetError: 72224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.685814] LNet: Removed LNI 192.168.201.6@tcp [ 866.899573] Key type .llcrypt unregistered [ 866.901840] Key type ._llcrypt unregistered [ 876.579943] Key type ._llcrypt registered [ 876.581613] Key type .llcrypt registered [ 877.337148] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:20:54 (1788891654) [ 877.725753] Key type .llcrypt unregistered [ 877.727390] Key type ._llcrypt unregistered [ 880.905484] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 880.938677] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 880.941356] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 880.945138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 883.230559] Key type ._llcrypt registered [ 883.232293] Key type .llcrypt registered [ 883.356937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.367331] alg: No test for adler32 (adler32-zlib) [ 884.230555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.267703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 884.285254] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 884.732425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 884.756857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.438805] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:21:06 (1788891666) [ 889.559317] LNet: 74288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.562496] LNetError: 74288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 889.564949] LNet: Removed LNI 10.1.2.3@tcp [ 889.571392] LNet: Removed LNI 192.168.201.6@tcp [ 889.871362] Key type .llcrypt unregistered [ 889.873368] Key type ._llcrypt unregistered [ 893.717682] Key type ._llcrypt registered [ 893.721091] Key type .llcrypt registered [ 893.845180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 893.854181] alg: No test for adler32 (adler32-zlib) [ 894.697466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.734293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 894.748118] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 895.087974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 895.096853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.537572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.6@tcp [ 896.607128] LNetError: Unexpected error -22 connecting to 192.168.201.6@tcp at host 192.168.201.6:988 [ 898.541727] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:21:16 (1788891676) [ 898.619895] LNet: 75410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.624841] LNetError: 75410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 898.631066] LNet: Removed LNI 10.1.2.3@tcp [ 898.639571] LNet: Removed LNI 192.168.201.6@tcp [ 898.970486] Key type .llcrypt unregistered [ 898.972329] Key type ._llcrypt unregistered [ 902.092210] Key type ._llcrypt registered [ 902.093591] Key type .llcrypt registered [ 902.203780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.212861] alg: No test for adler32 (adler32-zlib) [ 903.050365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.100621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 903.122756] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 903.569664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 903.587270] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 904.164958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.6@tcp1 --nid 192.168.201.6@tcp2 [ 906.867908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp1 192.168.201.6@tcp2 [ 906.879509] LNetError: 76247:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.6@tcp1 to 192.168.201.6@tcp2 - Route table may be misconfigured [ 906.897771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp2 192.168.201.6@tcp1 [ 910.085786] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:21:27 (1788891687) [ 910.179721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.192651] LNet: 76821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.199401] LNetError: 76821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.201703] LNet: Removed LNI 192.168.201.6@tcp1 [ 911.276595] LNet: Removed LNI 192.168.201.6@tcp2 [ 911.300571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.337506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 911.351114] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 911.676768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 911.689976] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 912.087780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 912.108463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.6@tcp [ 912.125571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp1 192.168.201.6@tcp1 [ 912.152686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.6@tcp -d 192.168.201.6@tcp -e network_timeout [ 912.172483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -e network_timeout [ 912.197663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -e network_timeout [ 912.224572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -e network_timeout [ 912.260609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 912.275303] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 912.292853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 912.351457] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 912.371704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 912.428285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 915.619235] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:21:33 (1788891693) [ 915.714438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.724564] LNet: 77682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.729305] LNetError: 77682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.732127] LNetError: 77682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 915.735488] LNet: Removed LNI 192.168.201.6@tcp [ 916.773592] LNet: Removed LNI 192.168.201.6@tcp1 [ 916.787368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.835564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 916.964835] LNet: 77720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.158866] Key type .llcrypt unregistered [ 917.160582] Key type ._llcrypt unregistered [ 921.905239] Key type ._llcrypt registered [ 921.906796] Key type .llcrypt registered [ 922.489648] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:21:40 (1788891700) [ 922.630664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 922.639349] alg: No test for adler32 (adler32-zlib) [ 923.454541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.095929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 924.881310] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:21:42 (1788891702) [ 924.988911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 924.998274] LNet: 78813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.016096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.050240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.063394] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 925.378746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 925.389705] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 925.756872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 925.773770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp1 [ 925.793796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp1 [ 928.496109] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:21:46 (1788891706) [ 929.447505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg106-server [ 930.357833] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:21:47 (1788891707) [ 931.293704] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg106-server [ 932.033168] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:21:49 (1788891709) [ 932.895856] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg106-server [ 933.701261] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:21:51 (1788891711) [ 934.574763] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg106-server [ 935.501898] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:21:53 (1788891713) [ 936.334310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg106-server [ 937.147562] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:21:54 (1788891714) [ 938.022119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg106-server [ 938.926453] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:21:56 (1788891716) [ 939.808926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg106-server [ 940.843290] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:21:58 (1788891718) [ 941.779438] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg106-server [ 942.735650] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:22:00 (1788891720) [ 943.708619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg106-server [ 944.586600] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:22:02 (1788891722) [ 944.701500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.713413] LNet: 82686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.717763] LNetError: 82686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.721705] LNet: Removed LNI 192.168.201.6@tcp [ 945.768119] LNet: Removed LNI 192.168.201.6@tcp1 [ 945.781302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.818777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 945.832749] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 946.198389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 946.238299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 946.311050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.322433] LNet: 82833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.325970] LNetError: 82833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.331705] LNetError: 82833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 946.340485] LNet: Removed LNI 192.168.201.6@tcp [ 946.349756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.386540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 946.399521] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 946.789596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 946.893405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.942182] LNet: Removed LNI 192.168.201.6@tcp [ 947.955655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.992410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 948.009635] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 948.364196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 948.397473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 948.495507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.511598] LNet: 83126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.514575] LNet: 83126:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 948.517765] LNetError: 83126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.521486] LNetError: 83126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 948.528493] LNet: Removed LNI 192.168.201.6@tcp [ 948.538553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.571388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 948.590036] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 948.990560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 949.097158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.187867] LNet: Removed LNI 192.168.201.6@tcp [ 950.208713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.246723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 950.267083] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 950.616937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 950.654540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 950.766698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 950.777960] LNetError: 83418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.781620] LNetError: 83418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 950.796044] LNet: Removed LNI 192.168.201.6@tcp [ 950.807498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 950.832370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 950.844685] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 951.216579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 951.333551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.356766] LNet: Removed LNI 192.168.201.6@tcp [ 952.380284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.424966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.441801] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 952.871048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 952.918141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 953.023429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 953.035261] LNet: 83711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.038758] LNet: 83711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 954.086364] LNet: Removed LNI 192.168.201.6@tcp [ 954.094873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 954.120764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 954.136113] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 954.519457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 955.692343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.705350] LNetError: 83865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.709517] LNetError: 83865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 955.718918] LNet: Removed LNI 192.168.201.6@tcp [ 955.730267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.758425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.780215] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 956.128191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 956.164441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 956.265592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.285896] LNet: Removed LNI 192.168.201.6@tcp [ 957.300479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.330139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 957.343879] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 957.707322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 957.811366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.885615] LNet: Removed LNI 192.168.201.6@tcp [ 958.895670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.919849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 958.931202] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 959.227981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 959.259465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 959.384113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.421545] LNet: Removed LNI 192.168.201.6@tcp [ 960.434262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.469624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 960.486070] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 960.869246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 960.985788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.021617] LNet: Removed LNI 192.168.201.6@tcp [ 962.034726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.061467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.075771] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 962.412290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 962.440492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.450149] LNet: 84582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.452929] LNet: 84582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 962.467554] LNet: Removed LNI 192.168.201.6@tcp [ 962.477396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.504804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.519955] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 962.861919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 962.897717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 962.920451] LNet: Removed LNI 192.168.201.6@tcp [ 962.933166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.961747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 962.999111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.016075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.039519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 963.081462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.102807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.128204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 963.163199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.178463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.204440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 963.216990] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 963.562123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 963.579476] LNet: Removed LNI 192.168.201.6@tcp [ 963.587929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.617139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 966.899371] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:22:24 (1788891744) [ 966.971669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.992631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.022888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 967.041549] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/180] [ 967.657973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.670280] LNetError: 85608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 967.673491] LNetError: 85608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 967.684916] LNet: Removed LNI 192.168.201.6@tcp231 [ 967.694187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.732233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 967.745576] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/180] [ 967.748564] LNet: 85616:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.949625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 967.978779] LNet: Removed LNI 192.168.201.6@tcp231 [ 967.989237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 968.002875] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/180] [ 968.556734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.572494] LNet: Removed LNI 192.168.201.6@tcp231 [ 968.585419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.623305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 968.640285] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/0] [ 968.646220] LNet: 85768:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 968.839587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 968.864390] LNet: Removed LNI 192.168.201.6@tcp231 [ 968.876850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 968.890067] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/0] [ 969.421220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.441375] LNet: Removed LNI 192.168.201.6@tcp231 [ 969.454448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.487623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 969.503055] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/60] [ 969.677202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 969.694778] LNet: Removed LNI 192.168.201.6@tcp231 [ 969.704907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 969.719406] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/60] [ 970.209878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.227458] LNet: Removed LNI 192.168.201.6@tcp231 [ 970.241767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.279248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 970.297294] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/180] [ 970.300465] LNet: 86070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.304420] LNet: 86070:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 970.498401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 970.519980] LNet: Removed LNI 192.168.201.6@tcp231 [ 970.529910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 970.545868] LNet: Added LNI 192.168.201.6@tcp231 [8/256/0/180] [ 973.882621] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:22:31 (1788891751) [ 973.999404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.020730] LNet: Removed LNI 192.168.201.6@tcp231 [ 974.032310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.066441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 974.077127] LNet: Added LNI 192.168.201.6@tcp232 [8/256/0/180] [ 974.428162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 974.448836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 974.471342] LNet: Removed LNI 192.168.201.6@tcp232 [ 974.478873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 974.490736] LNet: Added LNI 192.168.201.6@tcp232 [8/256/0/180] [ 974.830021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 974.849973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 974.870950] LNet: Removed LNI 192.168.201.6@tcp232 [ 974.881309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 974.891314] LNet: Added LNI 192.168.201.6@tcp232 [8/256/0/180] [ 975.209238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 975.228598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 975.245554] LNet: Removed LNI 192.168.201.6@tcp232 [ 975.254697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 975.271747] LNet: Added LNI 192.168.201.6@tcp232 [8/256/0/180] [ 975.608064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 975.627457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 975.647260] LNet: Removed LNI 192.168.201.6@tcp232 [ 975.655178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 978.663946] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:22:36 (1788891756) [ 978.988143] LNet: 87906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.990550] LNet: 87906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 979.258664] Key type .llcrypt unregistered [ 979.263387] Key type ._llcrypt unregistered [ 982.271218] Key type ._llcrypt registered [ 982.273072] Key type .llcrypt registered [ 982.372109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.380191] alg: No test for adler32 (adler32-zlib) [ 983.219468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 983.235959] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 987.009469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 991.011413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.051408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 991.066124] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 991.875729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_interrupt -r 2 [ 991.895106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_interrupt -r 2 [ 991.917933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_interrupt -r 2 [ 991.940474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_interrupt -r 2 [ 991.959273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_interrupt -r 2 [ 991.984986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_interrupt -r 2 [ 992.004917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_interrupt -r 2 [ 992.024217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_interrupt -r 2 [ 992.043224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.106@tcp [ 992.059586] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 992.063654] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 992.189230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.213558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 992.233613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.259719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.277568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.296652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.314752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.333729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.345754] LNet: 88564:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.359332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.383287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.405136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.419598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.438558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.459288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.766310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_dropped -r 2 [ 992.783719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_dropped -r 2 [ 992.802987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_dropped -r 2 [ 992.824531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_dropped -r 2 [ 992.852146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_dropped -r 2 [ 992.880424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_dropped -r 2 [ 992.906155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_dropped -r 2 [ 992.935242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_dropped -r 2 [ 992.954434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.106@tcp [ 992.962813] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 993.084343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.105481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 993.136466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.154728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.172263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.188311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 993.207878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 993.232794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.242881] LNet: 88684:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 993.252468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.268374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.286089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 993.299899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 993.315908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.329164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 993.541492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_aborted -r 2 [ 993.570743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_aborted -r 2 [ 993.589765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_aborted -r 2 [ 993.610628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_aborted -r 2 [ 993.632424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_aborted -r 2 [ 993.663979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_aborted -r 2 [ 993.687724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_aborted -r 2 [ 993.713170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_aborted -r 2 [ 993.733354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.106@tcp [ 993.842162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.857426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 993.877197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.893201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.912961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.927846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 993.942308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 993.956460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.970886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.984609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.000830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.018393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.036285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.059091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.256274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_no_route -r 2 [ 994.273088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_no_route -r 2 [ 994.289293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_no_route -r 2 [ 994.305852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_no_route -r 2 [ 994.324543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_no_route -r 2 [ 994.343037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_no_route -r 2 [ 994.364649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_no_route -r 2 [ 994.386573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_no_route -r 2 [ 994.406371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.106@tcp [ 994.420485] LNet: There was an unexpected network error while writing to 192.168.201.106: rc = -22 [ 994.422781] LNet: Skipped 2 previous similar messages [ 994.537729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.551877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 994.566409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.582205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.597672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.612954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.629175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.646828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.662394] LNet: 88924:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 994.665767] LNet: 88924:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 994.674895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.692994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.711875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.726667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.750574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.767725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.986041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e local_timeout -r 2 [ 995.001476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e local_timeout -r 2 [ 995.020949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e local_timeout -r 2 [ 995.037385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e local_timeout -r 2 [ 995.058780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e local_timeout -r 2 [ 995.076189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e local_timeout -r 2 [ 995.092838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e local_timeout -r 2 [ 995.109913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e local_timeout -r 2 [ 995.128886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.106@tcp [ 995.248652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.263958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 995.279835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.296204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.315374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.330163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 995.347670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 995.369843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 995.385240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 995.401383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.415335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.429647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.444545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.461471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.669653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp -m GET -e remote_dropped -r 2 [ 995.689439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp -d 192.168.201.106@tcp1 -m GET -e remote_dropped -r 2 [ 995.706219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp -m GET -e remote_dropped -r 2 [ 995.722049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp -d 192.168.201.6@tcp1 -m GET -e remote_dropped -r 2 [ 995.740952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp -m GET -e remote_dropped -r 2 [ 995.760816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.106@tcp1 -m GET -e remote_dropped -r 2 [ 995.779644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp -m GET -e remote_dropped -r 2 [ 995.795654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.6@tcp1 -d 192.168.201.6@tcp1 -m GET -e remote_dropped -r 2 [ 995.812165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.6@tcp 192.168.201.106@tcp [ 995.823177] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.106@tcp [ 995.905598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 995.925236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 995.938936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 995.954263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 995.969652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 995.985274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.003024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.018405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.035464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.051887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.070526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.085421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.111928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.134726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 998.732840] LNet: 89237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.736239] LNetError: 89237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.740255] LNet: Removed LNI 192.168.201.6@tcp [ 998.745506] LNet: Removed LNI 192.168.201.6@tcp1 [ 998.932677] Key type .llcrypt unregistered [ 998.934434] Key type ._llcrypt unregistered [ 1006.624388] Key type ._llcrypt registered [ 1006.628674] Key type .llcrypt registered [ 1007.280234] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:23:04 (1788891784) [ 1007.499816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1007.507425] alg: No test for adler32 (adler32-zlib) [ 1008.338165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.367841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1008.383210] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1008.767311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1008.786408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.6@tcp [ 1008.795955] LNetError: 90211:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.6@tcp to recovery [ 1008.832190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.6@tcp [ 1008.876094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1011.753739] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:23:09 (1788891789) [ 1011.802241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.814758] LNet: 90794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.818498] LNetError: 90794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.832373] LNet: Removed LNI 192.168.201.6@tcp [ 1011.844938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.878621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1011.892213] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1012.237256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1012.248274] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 1012.679344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.6@tcp [ 1012.687812] LNetError: 91066:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.6@tcp to recovery [ 1012.895373] LNet: 1 local NIs in recovery (showing 1): 192.168.201.6@tcp [ 1012.896198] LNet: There was an unexpected network error while writing to 192.168.201.6: rc = -22 [ 1013.788674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.6@tcp [ 1018.779873] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:23:16 (1788891796) [ 1019.649500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg106-server [ 1020.555899] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:23:18 (1788891798) [ 1021.446908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg106-server [ 1022.207124] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:23:19 (1788891799) [ 1023.066781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg106-server [ 1023.852393] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:23:21 (1788891801) [ 1023.890730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.900746] LNet: 92687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.903344] LNetError: 92687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.905985] LNet: Removed LNI 192.168.201.6@tcp [ 1023.912776] LNet: Removed LNI 192.168.201.6@tcp1 [ 1023.924227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.967420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1023.981854] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1024.400027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1024.442378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1024.464549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1024.526965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1024.549652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1024.598249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1027.377701] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:23:24 (1788891804) [ 1027.426291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.439679] LNet: 93420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1027.444375] LNetError: 93420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1027.449903] LNetError: 93420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1027.461850] LNet: Removed LNI 192.168.201.6@tcp [ 1027.473650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.493808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.502453] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1030.532836] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:23:28 (1788891808) [ 1030.618449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.633261] LNet: 94128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.637470] LNetError: 94128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.648876] LNet: Removed LNI 192.168.201.6@tcp [ 1030.669717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.696667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1030.706532] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1034.052582] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:23:31 (1788891811) [ 1034.721977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1035.545904] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:23:33 (1788891813) [ 1035.928115] LNet: 95162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.932259] LNetError: 95162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.950720] LNet: Removed LNI 192.168.201.6@tcp [ 1036.351745] Key type .llcrypt unregistered [ 1036.353560] Key type ._llcrypt unregistered [ 1039.225612] Key type ._llcrypt registered [ 1039.227379] Key type .llcrypt registered [ 1039.304183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1039.310538] alg: No test for adler32 (adler32-zlib) [ 1040.129118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1040.142602] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1044.382694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 1046.430387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.106@tcp [ 1046.591283] LNet: 95575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.594991] LNetError: 95575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.610944] LNet: Removed LNI 192.168.201.6@tcp [ 1046.806714] Key type .llcrypt unregistered [ 1046.809279] Key type ._llcrypt unregistered [ 1051.616655] Key type ._llcrypt registered [ 1051.618891] Key type .llcrypt registered [ 1052.167880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1052.828125] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1053.494408] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:23:51 (1788891831) [ 1054.133022] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1054.808120] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:23:52 (1788891832) [ 1055.601563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg106-server [ 1056.283740] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:23:54 (1788891834) [ 1057.008217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg106-server [ 1057.736912] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:23:55 (1788891835) [ 1058.285283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1058.334987] alg: No test for adler32 (adler32-zlib) [ 1059.225199] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1059.329653] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1060.919176] Key type lgssc registered [ 1061.447897] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1070.749100] Key type lgssc unregistered [ 1070.883513] LNet: 98740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.886545] LNetError: 98740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.896657] LNet: Removed LNI 192.168.201.6@tcp [ 1071.185585] Key type .llcrypt unregistered [ 1071.187671] Key type ._llcrypt unregistered [ 1077.011912] Key type ._llcrypt registered [ 1077.013454] Key type .llcrypt registered [ 1077.510773] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:24:15 (1788891855) [ 1077.681868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1077.688531] alg: No test for adler32 (adler32-zlib) [ 1078.498852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.528515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1078.547545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1078.566606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1078.584810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1080.988438] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:24:18 (1788891858) [ 1081.042349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.054212] LNet: 100150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.065392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.092858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1081.104869] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1081.459207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.469038] LNetError: 100295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.481437] LNet: Removed LNI 192.168.201.6@tcp [ 1081.489822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.512236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1081.523669] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1081.703421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.712864] LNet: 100323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.715838] LNet: 100323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1081.726439] LNet: Removed LNI 192.168.201.6@tcp [ 1081.735358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.765658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1081.777749] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1084.433526] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:24:22 (1788891862) [ 1084.490367] LNet: 100910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.493849] LNetError: 100910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.497717] LNetError: 100910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1084.508262] LNet: Removed LNI 192.168.201.6@tcp [ 1084.779759] Key type .llcrypt unregistered [ 1084.781048] Key type ._llcrypt unregistered [ 1087.327650] Key type ._llcrypt registered [ 1087.329133] Key type .llcrypt registered [ 1087.427578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.434811] alg: No test for adler32 (adler32-zlib) [ 1088.251376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.285107] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1088.286618] LNet: *** cfs_fail_loc=e003, val=100*** [ 1088.590506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1088.602955] Ping reply received with mapping: [ 1088.604690] found metadata reply NI[192.168.201.6@tcp] [ 1088.606374] 06 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1088.608934] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1088.611206] 5a 5a 5a 5a 5a 5a 5a 5a [ 1088.676104] LNet: 101306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.679908] LNetError: 101306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.696864] LNet: Removed LNI 192.168.201.6@tcp [ 1088.968503] Key type .llcrypt unregistered [ 1088.970221] Key type ._llcrypt unregistered [ 1093.254614] Key type ._llcrypt registered [ 1093.256292] Key type .llcrypt registered [ 1093.780708] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:24:31 (1788891871) [ 1094.008532] Key type .llcrypt unregistered [ 1094.009887] Key type ._llcrypt unregistered [ 1096.509708] Key type ._llcrypt registered [ 1096.512474] Key type .llcrypt registered [ 1096.630419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.638815] alg: No test for adler32 (adler32-zlib) [ 1097.525296] LNetError: 102341:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1097.696749] Key type .llcrypt unregistered [ 1097.698236] Key type ._llcrypt unregistered [ 1099.718509] Key type ._llcrypt registered [ 1099.719686] Key type .llcrypt registered [ 1100.232904] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:24:37 (1788891877) [ 1100.478231] Key type .llcrypt unregistered [ 1100.479564] Key type ._llcrypt unregistered [ 1102.945879] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1102.949186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.159637] Key type ._llcrypt registered [ 1105.160820] Key type .llcrypt registered [ 1105.243522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1105.251194] alg: No test for adler32 (adler32-zlib) [ 1106.070939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.103670] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1110.648455] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:24:48 (1788891888) [ 1110.707093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1110.719144] LNet: 104112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1110.722468] LNetError: 104112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1110.732534] LNet: Removed LNI 10.1.2.3@tcp [ 1110.743113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.032539] Key type .llcrypt unregistered [ 1111.034784] Key type ._llcrypt unregistered [ 1113.441425] Key type ._llcrypt registered [ 1113.442607] Key type .llcrypt registered [ 1113.517743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.523763] alg: No test for adler32 (adler32-zlib) [ 1114.348290] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1114.377990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1114.439994] LNet: 104397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.443660] LNetError: 104397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.460716] LNet: Removed LNI 192.168.201.6@tcp [ 1114.708755] Key type .llcrypt unregistered [ 1114.710550] Key type ._llcrypt unregistered [ 1119.132569] Key type ._llcrypt registered [ 1119.133680] Key type .llcrypt registered [ 1119.732561] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:24:57 (1788891897) [ 1119.972294] Key type .llcrypt unregistered [ 1119.974063] Key type ._llcrypt unregistered [ 1122.313028] Key type ._llcrypt registered [ 1122.314459] Key type .llcrypt registered [ 1122.384155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1122.390496] alg: No test for adler32 (adler32-zlib) [ 1125.666192] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:25:03 (1788891903) [ 1125.734630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1125.754335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1125.787883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1125.806875] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1126.135419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1126.150634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1126.168909] LNet: Removed LNI 192.168.201.6@tcp [ 1126.274905] LNet: 106133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1126.426585] Key type .llcrypt unregistered [ 1126.429990] Key type ._llcrypt unregistered [ 1130.667132] Key type ._llcrypt registered [ 1130.668260] Key type .llcrypt registered [ 1131.247748] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:25:08 (1788891908) [ 1131.424166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1131.429947] alg: No test for adler32 (adler32-zlib) [ 1132.243526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.271468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1132.283957] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1132.595066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1132.612785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.6@tcp [ 1132.623363] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1132.625594] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1132.627444] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1132, 0, 0 [ 1132.630593] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1132.632480] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1132.635853] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1132.637892] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1132, 0, 0 [ 1132.640858] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1132.642733] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1132.644833] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1132.647213] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1132, 0, 1182 [ 1132.652584] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1132.654192] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1132.656887] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1132.659226] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1132, 0, 1182 [ 1132.662133] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1135.366368] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:25:13 (1788891913) [ 1135.576664] LNet: 107667:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.579529] LNetError: 107667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.587686] LNet: Removed LNI 192.168.201.6@tcp [ 1135.867028] Key type .llcrypt unregistered [ 1135.868436] Key type ._llcrypt unregistered [ 1138.304965] Key type ._llcrypt registered [ 1138.306431] Key type .llcrypt registered [ 1138.384352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.390390] alg: No test for adler32 (adler32-zlib) [ 1139.205731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1139.219227] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1142.789835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 1146.369665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1146.385517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1146.395654] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 1146.852918] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1146.952149] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1146.955566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1146.958690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1149.093658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1149.103444] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1153.292289] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:25:30 (1788891930) [ 1153.365518] LNet: 109166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1153.369055] LNetError: 109166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1153.372371] LNet: Removed LNI 192.168.201.6@tcp [ 1154.399294] LNet: Removed LNI 192.168.201.6@tcp1 [ 1154.401340] LNetError: 109166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1154.404826] LNetError: 109166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1154.414494] LNet: Removed LNI 10.1.2.3@tcp99 [ 1154.676791] Key type .llcrypt unregistered [ 1154.678526] Key type ._llcrypt unregistered [ 1157.325813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1157.328847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1159.553815] Key type ._llcrypt registered [ 1159.555127] Key type .llcrypt registered [ 1159.635651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.648335] alg: No test for adler32 (adler32-zlib) [ 1160.474092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1160.502414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1160.514228] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1160.852784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1160.863360] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1161.813261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1161.838572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1161.867385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1161.880425] LNetError: 109922:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1161.900427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1161.920935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1162.115857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1162.158520] LNet: 109942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1162.161870] LNetError: 109942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1162.165189] LNet: Removed LNI 10.1.2.3@tcp [ 1163.178408] LNet: Removed LNI 192.168.201.6@tcp [ 1163.432923] Key type .llcrypt unregistered [ 1163.435095] Key type ._llcrypt unregistered [ 1167.962734] Key type ._llcrypt registered [ 1167.963950] Key type .llcrypt registered [ 1168.584680] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:25:46 (1788891946) [ 1168.788958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1168.797897] alg: No test for adler32 (adler32-zlib) [ 1169.626620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.657773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1169.674613] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1170.003199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1170.045575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg106-client@tcp [ 1172.588078] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:25:50 (1788891950) [ 1172.660335] LNet: 111469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.663723] LNetError: 111469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.671818] LNet: Removed LNI 192.168.201.6@tcp [ 1172.931862] Key type .llcrypt unregistered [ 1172.933227] Key type ._llcrypt unregistered [ 1175.270378] Key type ._llcrypt registered [ 1175.271434] Key type .llcrypt registered [ 1175.345742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1175.352583] alg: No test for adler32 (adler32-zlib) [ 1176.171992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1176.185232] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1176.199458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1178.957862] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:25:56 (1788891956) [ 1179.572357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1180.281633] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:25:57 (1788891957) [ 1180.963415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1181.721434] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:25:59 (1788891959) [ 1181.754524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1181.771342] LNet: 112882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.773687] LNetError: 112882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1181.783879] LNet: Removed LNI 192.168.201.6@tcp [ 1181.797163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1181.835987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1181.852817] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1182.282281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1182.314315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1185.521792] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:26:03 (1788891963) [ 1185.618077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1185.632581] LNet: 113597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.636703] LNetError: 113597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.654475] LNet: Removed LNI 192.168.201.6@tcp [ 1185.669280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1185.692726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1185.704943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1186.014310] Key type .llcrypt unregistered [ 1186.015769] Key type ._llcrypt unregistered [ 1190.856941] Key type ._llcrypt registered [ 1190.858359] Key type .llcrypt registered [ 1191.461219] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:26:09 (1788891969) [ 1191.621970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1191.629264] alg: No test for adler32 (adler32-zlib) [ 1192.440508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.464527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1192.475953] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1192.789768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1192.819799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1195.525253] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:26:13 (1788891973) [ 1195.559542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1195.570718] LNet: 115141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.572882] LNetError: 115141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.586735] LNet: Removed LNI 192.168.201.6@tcp [ 1195.602424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1195.627347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1195.645800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1198.306783] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:26:15 (1788891975) [ 1198.378407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1198.390493] LNet: 115727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1198.402465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.431497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1198.598785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1198.613929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1198.641398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1198.794888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1198.809366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1198.842633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1198.860804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1198.891126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1199.039517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1199.055635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1199.079187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1199.234428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1199.253837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1202.137337] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:26:19 (1788891979) [ 1202.224709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1202.241199] LNet: 116358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.243856] LNet: 116358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1202.260291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1202.291227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1202.305384] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1202.656365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1202.669366] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1203.082079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1203.111237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1203.210718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1203.226136] LNetError: 116639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.229761] LNet: Removed LNI 192.168.201.6@tcp [ 1203.235846] LNet: Removed LNI 192.168.201.6@tcp2 [ 1203.252946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1203.281386] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1203.284600] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1203.287203] LNet: 116645:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1203.499927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1203.605374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1204.639116] LNet: Removed LNI 192.168.201.6@tcp [ 1204.641139] LNetError: 116682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1204.645599] LNetError: 116682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1204.655522] LNet: Removed LNI 192.168.201.6@tcp2 [ 1204.679759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1204.716625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1204.734098] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1205.074596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1205.085186] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1205.521145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1205.543607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1205.571489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1205.680922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1205.697843] LNet: 116966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.701727] LNet: 116966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1206.751667] LNet: Removed LNI 192.168.201.6@tcp [ 1206.755148] LNetError: 116966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.758478] LNetError: 116966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1206.773926] LNet: Removed LNI 192.168.201.6@tcp2 [ 1206.785894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1206.818494] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1206.824080] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1206.828292] LNet: 116972:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1207.009754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1207.105176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1207.120028] LNet: Removed LNI 192.168.201.6@tcp [ 1208.165524] LNet: Removed LNI 192.168.201.6@tcp2 [ 1208.182391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1208.208186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1208.225541] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1208.544268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1208.555223] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1208.921981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1208.948598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1209.045912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.079175] LNet: Removed LNI 192.168.201.6@tcp [ 1210.087440] LNet: Removed LNI 192.168.201.6@tcp2 [ 1210.097536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.133611] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1210.138663] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1210.142254] LNet: 117295:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.342908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1210.449387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.460810] LNet: 117332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.464268] LNet: 117332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1211.487169] LNet: Removed LNI 192.168.201.6@tcp [ 1211.488852] LNetError: 117332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1211.492066] LNetError: 117332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1211.500171] LNet: Removed LNI 192.168.201.6@tcp2 [ 1211.510472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1211.536455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1211.550525] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1211.884051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1211.895856] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1212.310654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1212.331750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1212.357706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1212.468277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1213.535387] LNet: Removed LNI 192.168.201.6@tcp [ 1214.564616] LNet: Removed LNI 192.168.201.6@tcp2 [ 1214.578178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1214.605502] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1214.609317] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1214.612738] LNet: 117623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.820855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1214.902471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1214.912625] LNet: Removed LNI 192.168.201.6@tcp [ 1215.973439] LNet: Removed LNI 192.168.201.6@tcp2 [ 1215.991154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.017806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1216.030533] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1216.364049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1216.374874] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1216.769445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1216.787276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1216.816097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1216.991980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1218.015523] LNet: Removed LNI 192.168.201.6@tcp [ 1218.024841] LNet: Removed LNI 192.168.201.6@tcp2 [ 1218.035366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1218.064460] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1218.071763] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1218.274719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1218.406876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1219.424125] LNet: Removed LNI 192.168.201.6@tcp [ 1219.433418] LNet: Removed LNI 192.168.201.6@tcp2 [ 1219.448752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.480964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1219.492568] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1219.837161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1219.849270] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1220.305878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1220.325968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1220.341336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1220.369725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1220.546250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1220.560811] LNet: 118316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1220.564513] LNet: 118316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1220.569240] LNetError: 118316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1220.572270] LNetError: 118316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1220.575329] LNet: Removed LNI 192.168.201.6@tcp [ 1221.611635] LNet: Removed LNI 192.168.201.6@tcp2 [ 1221.627242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.678337] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1221.683380] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1221.685957] LNet: 118322:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.688018] LNet: 118322:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1221.923238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1225.084424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1225.831135] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:26:43 (1788892003) [ 1225.872501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1225.892791] LNet: Removed LNI 192.168.201.6@tcp [ 1226.920550] LNet: Removed LNI 192.168.201.6@tcp2 [ 1226.932971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.054573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1227.068329] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1227.471179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1227.482669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1227.896701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1227.910080] LNet: Removed LNI 10.1.2.3@tcp [ 1227.921316] LNet: Removed LNI 192.168.201.6@tcp [ 1227.932328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.959918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1227.970328] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1227.972483] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1228.135742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1228.147562] LNet: Removed LNI 192.168.201.6@tcp [ 1228.157192] LNet: Removed LNI 10.1.2.3@tcp [ 1230.854682] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 14:26:48 (1788892008) [ 1230.887494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1230.914662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.946896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1230.960220] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1231.268284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1231.293554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.312136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.330259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.353717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.370426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.387270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.402165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.416732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.436473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.454164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.6@tcp [ 1231.534551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1234.453649] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:26:52 (1788892012) [ 1234.533824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1235.557561] LNet: Removed LNI 192.168.201.6@tcp [ 1235.567445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.613335] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1235.665739] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1235.667605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1235.670024] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1237.807644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1237.818736] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1238.104083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1238.114593] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1239.006346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1239.026358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1239.066826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.6@tcp -m PUT -e local_timeout -r 1 [ 1239.103345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1239.134345] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1239.136073] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1239.236741] LNetError: 120751:0:(lib-move.c:4801:lnet_parse()) 192.168.201.6@tcp, src 192.168.201.6@tcp: Dropping PUT (error -108 looking up sender) [ 1240.287466] LNet: 121210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.291314] LNet: 121210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1240.295370] LNetError: 121210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1240.298718] LNetError: 121210:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1240.302348] LNet: Removed LNI 10.1.2.3@tcp [ 1241.317419] LNet: Removed LNI 192.168.201.6@tcp [ 1241.476092] Key type .llcrypt unregistered [ 1241.478682] Key type ._llcrypt unregistered [ 1246.010652] Key type ._llcrypt registered [ 1246.011562] Key type .llcrypt registered [ 1246.566689] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:27:04 (1788892024) [ 1246.706047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1246.712742] alg: No test for adler32 (adler32-zlib) [ 1247.522246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.685693] LNet: 122058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.952801] Key type .llcrypt unregistered [ 1247.954242] Key type ._llcrypt unregistered [ 1250.449564] Key type ._llcrypt registered [ 1250.450947] Key type .llcrypt registered [ 1250.530841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1250.536743] alg: No test for adler32 (adler32-zlib) [ 1251.350282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1251.367545] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1254.816446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 1254.899169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1255.156110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.106@tcp [ 1255.253601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.270191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.284106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.298950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.320756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.341125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.367457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.386174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.395210] LNet: 122503:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.403858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.419206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.436991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.448536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.456164] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.106@tcp [ 1255.463688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.481196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.751553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1255.855560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1255.870418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.885444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.900626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.917156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.932869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.952300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.966901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.977706] LNet: 122620:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.989355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.008271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.035309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.055782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.072402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.088973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.379266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.106@tcp [ 1256.492503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.505567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.521494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.536640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.552373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.569923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.587825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.610107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.624316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.642353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.661334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.678654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.696623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.713710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.002616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1257.133025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.152961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.168429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.183923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.201507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.214938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.229849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.244058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.252761] LNet: 122854:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.255422] LNet: 122854:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1257.265294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.284356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.302733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.318246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.343549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.361645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1257.510332] LNet: 122900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.514618] LNetError: 122900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.524562] LNet: Removed LNI 192.168.201.6@tcp [ 1257.689922] Key type .llcrypt unregistered [ 1257.692176] Key type ._llcrypt unregistered [ 1264.246111] Key type ._llcrypt registered [ 1264.247989] Key type .llcrypt registered [ 1264.859265] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:27:22 (1788892042) [ 1265.039961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1265.048666] alg: No test for adler32 (adler32-zlib) [ 1265.873525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.064828] LNet: 123783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1266.375567] Key type .llcrypt unregistered [ 1266.378175] Key type ._llcrypt unregistered [ 1269.021551] Key type ._llcrypt registered [ 1269.023315] Key type .llcrypt registered [ 1269.130683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.141837] alg: No test for adler32 (adler32-zlib) [ 1269.973162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1269.986359] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1273.649309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.106@tcp [ 1277.084571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.101549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1277.113275] LNet: Added LNI 192.168.201.6@tcp1 [8/256/0/180] [ 1277.525481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1277.785480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.106@tcp [ 1277.897891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.914266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.930268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.947904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.965382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.982278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.000556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.017695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.027771] LNet: 124429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.041475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.059076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.075271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.091735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.108772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.126176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.400241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1278.496534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.511653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1278.527721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.542909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.556928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.571941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.585794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.599917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.608602] LNet: 124547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.618141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.633210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.648121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.662607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.678466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.698053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.949109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.106@tcp [ 1279.040618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.056844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.071780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.086105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.101888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.119326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.135922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.151545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.165598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.181909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.195630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.200437] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.106@tcp [ 1279.210130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.225326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.243593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1279.525031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1279.624480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.637592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.654862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.669622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.683565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.703274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.717068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.732744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.740128] LNet: 124781:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.742918] LNet: 124781:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1279.752699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.769025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.783440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.798848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.813720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.828562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.394489] LNet: 124861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.399106] LNetError: 124861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.405394] LNet: Removed LNI 192.168.201.6@tcp [ 1282.419523] LNet: Removed LNI 192.168.201.6@tcp1 [ 1282.590784] Key type .llcrypt unregistered [ 1282.592312] Key type ._llcrypt unregistered [ 1289.067447] Key type ._llcrypt registered [ 1289.068634] Key type .llcrypt registered [ 1289.596647] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 14:27:47 (1788892067) [ 1289.746275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1289.753684] alg: No test for adler32 (adler32-zlib) [ 1290.562905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.585760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1290.595799] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1290.889049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1290.897970] LNet: Added LNI 192.168.201.6@tcp2 [8/256/0/180] [ 1291.273573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.6@tcp [ 1291.315930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1292.383152] LNet: Removed LNI 192.168.201.6@tcp [ 1294.984413] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 14:27:52 (1788892072) [ 1295.604533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1296.284644] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:27:53 (1788892073) [ 1299.054565] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:27:56 (1788892076) [ 1301.784871] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:27:59 (1788892079) [ 1304.609093] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:28:02 (1788892082) [ 1307.198067] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:28:04 (1788892084) [ 1309.881509] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:28:07 (1788892087) [ 1312.637066] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:28:10 (1788892090) [ 1315.511954] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:28:13 (1788892093) [ 1318.237951] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:28:16 (1788892096) [ 1320.800135] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:28:18 (1788892098) [ 1323.283939] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:28:21 (1788892101) [ 1325.842759] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:28:23 (1788892103) [ 1328.448927] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:28:26 (1788892106) [ 1331.035107] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:28:28 (1788892108) [ 1333.652595] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:28:31 (1788892111) [ 1336.205480] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:28:33 (1788892113) [ 1338.800905] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:28:36 (1788892116) [ 1341.507710] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:28:39 (1788892119) [ 1341.586535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1341.600552] LNet: 136536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.603826] LNetError: 136536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1341.616701] LNet: Removed LNI 192.168.201.6@tcp2 [ 1341.626348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.673380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1341.683156] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1342.123823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1342.133021] LNet: 136678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.135809] LNetError: 136678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.145399] LNet: Removed LNI 192.168.201.6@tcp [ 1342.154118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1342.178390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1342.188637] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1344.819475] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:28:42 (1788892122) [ 1347.457826] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:28:45 (1788892125) [ 1350.251927] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:28:47 (1788892127) [ 1353.027758] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:28:50 (1788892130) [ 1355.446688] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:28:53 (1788892133) [ 1358.137614] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:28:55 (1788892135) [ 1360.942443] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:28:58 (1788892138) [ 1364.132570] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:29:01 (1788892141) [ 1367.109814] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:29:04 (1788892144) [ 1369.899057] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:29:07 (1788892147) [ 1372.743692] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:29:10 (1788892150) [ 1375.334760] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:29:13 (1788892153) [ 1377.688964] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1264 sec ========== 14:29:15 (1788892155) [ 1377.783903] LNet: 144136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.787353] LNetError: 144136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.797535] LNet: Removed LNI 192.168.201.6@tcp [ 1378.033500] Key type .llcrypt unregistered [ 1378.034816] Key type ._llcrypt unregistered [ 1382.372281] Key type ._llcrypt registered [ 1382.373976] Key type .llcrypt registered [ 1382.547361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1382.553166] alg: No test for adler32 (adler32-zlib) [ 1383.420173] Lustre: Lustre: Build Version: 2.17.57_103_ga91bf53 [ 1383.519927] LNet: Added LNI 192.168.201.6@tcp [8/256/0/180] [ 1385.111151] Key type lgssc registered [ 1385.538099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1410.142954] Lustre: Mounted lustre-client - version 2.17.57_103_ga91bf53 [ 1412.591848] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1415.060381] Lustre: Unmounted lustre-client [ 1446.972633] Key type lgssc unregistered [ 1447.104432] LNet: 147075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.107279] LNetError: 147075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.118678] LNet: Removed LNI 192.168.201.6@tcp [ 1447.411134] Key type .llcrypt unregistered [ 1447.412523] Key type ._llcrypt unregistered