[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 461965581 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002209] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004007] kvm-guest: setup PV IPIs [ 0.006249] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009105] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012058] *** VALIDATE selinux *** [ 0.019601] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024181] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027123] *** VALIDATE tmpfs *** [ 0.029432] *** VALIDATE proc *** [ 0.030195] *** VALIDATE cgroup *** [ 0.031007] *** VALIDATE cgroup2 *** [ 0.032195] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034008] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036029] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.039752] debug: unmapping init [mem 0xffffffffab659000-0xffffffffab660fff] [ 0.041139] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042573] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043023] ... version: 2 [ 0.043993] ... bit width: 48 [ 0.044013] ... generic registers: 4 [ 0.045010] ... value mask: 0000ffffffffffff [ 0.046010] ... max period: 00007fffffffffff [ 0.047007] ... fixed-purpose events: 3 [ 0.047960] ... event mask: 000000070000000f [ 0.048255] rcu: Hierarchical SRCU implementation. [ 0.050190] smp: Bringing up secondary CPUs ... [ 0.051429] x86: Booting SMP configuration: [ 0.052022] .... node #0, CPUs: #1 #2 #3 [ 0.054760] smp: Brought up 1 node, 4 CPUs [ 0.056010] smpboot: Max logical packages: 1 [ 0.056808] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126552] node 0 deferred pages initialised in 68ms [ 0.129006] devtmpfs: initialized [ 0.130214] x86/mm: Memory block size: 128MB [ 0.132848] gcov: version magic: 0x41383552 [ 0.134084] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135063] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136222] pinctrl core: initialized pinctrl subsystem [ 0.137189] [ 0.137572] ************************************************************* [ 0.138010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139009] ** ** [ 0.140009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141009] ** ** [ 0.142009] ** This means that this kernel is built to expose internal ** [ 0.143013] ** IOMMU data structures, which may compromise security on ** [ 0.144009] ** your system. ** [ 0.145011] ** ** [ 0.146009] ** If you see this message and you are not debugging the ** [ 0.147015] ** kernel, report this immediately to your vendor! ** [ 0.148010] ** ** [ 0.149011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150010] ************************************************************* [ 0.151650] NET: Registered protocol family 16 [ 0.152447] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.155436] cpuidle: using governor menu [ 0.157995] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.160746] PCI: Using configuration type 1 for base access [ 0.163098] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175178] cryptd: max_cpu_qlen set to 1000 [ 0.178598] ACPI: Added _OSI(Module Device) [ 0.182020] ACPI: Added _OSI(Processor Device) [ 0.184014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187113] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194665] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200629] ACPI: Interpreter enabled [ 0.202064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.204014] ACPI: Using IOAPIC for interrupt routing [ 0.206105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223722] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.227049] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.230023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.234080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.239471] acpiphp: Slot [2] registered [ 0.241153] acpiphp: Slot [5] registered [ 0.242111] acpiphp: Slot [6] registered [ 0.244111] acpiphp: Slot [3] registered [ 0.246124] acpiphp: Slot [4] registered [ 0.247094] acpiphp: Slot [7] registered [ 0.249121] acpiphp: Slot [8] registered [ 0.250162] acpiphp: Slot [9] registered [ 0.252279] acpiphp: Slot [10] registered [ 0.253088] acpiphp: Slot [11] registered [ 0.254091] acpiphp: Slot [12] registered [ 0.256121] acpiphp: Slot [13] registered [ 0.257109] acpiphp: Slot [14] registered [ 0.259135] acpiphp: Slot [15] registered [ 0.260249] acpiphp: Slot [16] registered [ 0.262133] acpiphp: Slot [17] registered [ 0.263162] acpiphp: Slot [18] registered [ 0.266186] acpiphp: Slot [19] registered [ 0.268136] acpiphp: Slot [20] registered [ 0.270096] acpiphp: Slot [21] registered [ 0.271226] acpiphp: Slot [22] registered [ 0.273102] acpiphp: Slot [23] registered [ 0.274000] acpiphp: Slot [24] registered [ 0.275085] acpiphp: Slot [25] registered [ 0.276000] acpiphp: Slot [26] registered [ 0.278139] acpiphp: Slot [27] registered [ 0.279092] acpiphp: Slot [28] registered [ 0.281118] acpiphp: Slot [29] registered [ 0.283116] acpiphp: Slot [30] registered [ 0.284101] acpiphp: Slot [31] registered [ 0.286079] PCI host bridge to bus 0000:00 [ 0.288021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.290023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.295024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306101] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309598] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.319015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.323444] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.327021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335548] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338879] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341065] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.345416] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.353016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.365012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.368779] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.374809] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.400018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.410545] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.416027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.440015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.453875] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.457620] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.462897] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.465327] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.467209] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.471201] iommu: Default domain type: Passthrough [ 0.475130] SCSI subsystem initialized [ 0.477457] ACPI: bus type USB registered [ 0.479245] usbcore: registered new interface driver usbfs [ 0.481206] usbcore: registered new interface driver hub [ 0.483070] usbcore: registered new device driver usb [ 0.485183] pps_core: LinuxPPS API ver. 1 registered [ 0.487012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.491061] PTP clock support registered [ 0.493026] EDAC MC: Ver: 3.0.0 [ 0.495030] PCI: Using ACPI for IRQ routing [ 0.496658] NetLabel: Initializing [ 0.497010] NetLabel: domain hash size = 128 [ 0.499008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.501077] NetLabel: unlabeled traffic allowed by default [ 0.503221] vgaarb: loaded [ 0.504202] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.505008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.510406] clocksource: Switched to clocksource kvm-clock [ 0.608841] VFS: Disk quotas dquot_6.6.0 [ 0.610086] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.612036] *** VALIDATE ramfs *** [ 0.612788] *** VALIDATE hugetlbfs *** [ 0.613818] pnp: PnP ACPI init [ 0.615257] pnp: PnP ACPI: found 6 devices [ 0.629283] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.631983] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.633335] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.635079] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.636611] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.638090] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.639961] NET: Registered protocol family 2 [ 0.641946] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.647094] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.651317] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.655543] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.658052] TCP: Hash tables configured (established 65536 bind 65536) [ 0.660188] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.661962] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.664445] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.666551] NET: Registered protocol family 1 [ 0.668451] RPC: Registered named UNIX socket transport module. [ 0.669931] RPC: Registered udp transport module. [ 0.671049] RPC: Registered tcp transport module. [ 0.672463] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.674246] NET: Registered protocol family 44 [ 0.675842] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.678122] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.680142] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.683545] PCI: CLS 0 bytes, default 64 [ 0.686311] Unpacking initramfs... [ 2.226279] debug: unmapping init [mem 0xffff99247cc64000-0xffff99247ffcffff] [ 2.234862] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.238098] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.245990] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.521479] Initialise system trusted keyrings [ 4.523525] Key type blacklist registered [ 4.525926] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.553229] zbud: loaded [ 4.559264] *** VALIDATE nfs *** [ 4.561351] *** VALIDATE nfs4 *** [ 4.573174] pstore: using deflate compression [ 4.600193] Platform Keyring initialized [ 4.965740] NET: Registered protocol family 38 [ 4.969223] Key type asymmetric registered [ 4.976339] Asymmetric key parser 'x509' registered [ 4.980846] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.994031] io scheduler mq-deadline registered [ 5.001195] io scheduler kyber registered [ 5.011263] io scheduler bfq registered [ 5.019983] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.031031] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.038203] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.044320] ACPI: Power Button [PWRF] [ 5.055513] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.084834] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.150671] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.214660] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.317337] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.353547] Non-volatile memory driver v1.3 [ 5.357420] Linux agpgart interface v0.103 [ 5.509238] virtio_blk virtio1: [vda] 146048 512-byte logical blocks (74.8 MB/71.3 MiB) [ 5.527452] vda: detected capacity change from 0 to 74776576 [ 5.562538] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.568498] vdb: detected capacity change from 0 to 1073741824 [ 5.580495] libphy: Fixed MDIO Bus: probed [ 5.597423] usbcore: registered new interface driver usbserial_generic [ 5.605142] usbserial: USB Serial support registered for generic [ 5.613279] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.626246] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.647497] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.657564] mousedev: PS/2 mouse device common for all mice [ 5.665937] rtc_cmos 00:05: RTC can wake from S4 [ 5.686270] rtc_cmos 00:05: registered as rtc0 [ 5.686798] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.689699] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.697743] intel_pstate: CPU model not supported [ 5.726257] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.737058] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.742918] hid: raw HID events driver (C) Jiri Kosina [ 5.743746] usbcore: registered new interface driver usbhid [ 5.753024] usbhid: USB HID core driver [ 5.755739] drop_monitor: Initializing network drop monitor service [ 5.759262] Initializing XFRM netlink socket [ 5.762534] NET: Registered protocol family 10 [ 5.770469] Segment Routing with IPv6 [ 5.773733] NET: Registered protocol family 17 [ 5.781319] mpls_gso: MPLS GSO support [ 5.793954] RAS: Correctable Errors collector initialized. [ 5.798449] AVX version of gcm_enc/dec engaged. [ 5.806127] AES CTR mode by8 optimization enabled [ 6.154821] sched_clock: Marking stable (6154764906, 0)->(6984189276, -829424370) [ 6.160669] registered taskstats version 1 [ 6.163714] Loading compiled-in X.509 certificates [ 6.179094] zswap: loaded using pool lzo/zbud [ 6.295973] Key type big_key registered [ 6.353503] Key type encrypted registered [ 6.357857] ima: No TPM chip found, activating TPM-bypass! [ 6.364601] ima: Allocated hash algorithm: sha1 [ 6.372554] ima: No architecture policies found [ 6.377032] evm: Initialising EVM extended attributes: [ 6.379228] evm: security.selinux [ 6.380459] evm: security.ima [ 6.382734] evm: security.capability [ 6.385216] evm: HMAC attrs: 0x1 [ 6.394132] rtc_cmos 00:05: setting system clock to 2026-08-20 01:40:32 UTC (1787190032) [ 6.411577] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 6.421554] debug: unmapping init [mem 0xffffffffab382000-0xffffffffab658fff] [ 6.433098] Write protecting the kernel read-only data: 28672k [ 6.453241] debug: unmapping init [mem 0xffffffffa9a03000-0xffffffffa9bfffff] [ 6.465805] debug: unmapping init [mem 0xffffffffaa314000-0xffffffffaa3fffff] [ 6.554344] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.565402] systemd[1]: Detected virtualization kvm. [ 6.568196] systemd[1]: Detected architecture x86-64. [ 6.570781] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.617373] systemd[1]: No hostname configured. [ 6.620474] systemd[1]: Set hostname to . [ 6.624512] random: systemd: uninitialized urandom read (16 bytes read) [ 6.628703] systemd[1]: Initializing machine ID from random generator. [ 6.780270] random: ln: uninitialized urandom read (6 bytes read) [ 7.100707] random: systemd: uninitialized urandom read (16 bytes read) [ 7.111786] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 7.127839] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 7.141802] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.217424] device-mapper: uevent: version 1.0.3 [ 9.224439] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 11.141247] virtio_net virtio0 ens2: renamed from eth0 [ 11.361349] scsi host0: ata_piix [ 11.427267] scsi host1: ata_piix [ 11.437875] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.442960] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 16.902040] random: crng init done [ 16.905019] random: 7 urandom warning(s) missed due to ratelimiting [ 19.531401] dracut-initqueue[587]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 21.799553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.867063] printk: systemd: 26 output lines suppressed due to ratelimiting [ 25.951608] SELinux: Disabled at runtime. [ 26.066922] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.098877] systemd[1]: Detected virtualization kvm. [ 26.111373] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.611356] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.621163] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.636932] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.662628] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.669907] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.690179] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.714275] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ 28.826143] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 30.125796] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 31.060149] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 31.115444] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 31.370729] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 31.402030] EDAC sbridge: Ver: 1.1.2 [ 36.031591] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 37.243862] NFS: Registering the id_resolver key type [ 37.256091] Key type id_resolver registered [ 37.263501] Key type id_legacy registered [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg210-client login: [ 95.608078] hrtimer: interrupt took 4479046 ns [ 106.443466] libcfs: loading out-of-tree module taints kernel. [ 106.888327] Key type ._llcrypt registered [ 106.891050] Key type .llcrypt registered [ 107.407920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 107.426629] alg: No test for adler32 (adler32-zlib) [ 108.818285] Lustre: Lustre: Build Version: 2.17.57_44_gafaaa93 [ 110.083404] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 112.063188] Key type lgssc registered [ 114.155716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 254.061462] Lustre: Mounted lustre-client - version 2.17.57_44_gafaaa93 [ 259.289181] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 278.963275] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing check_logdir /tmp/testlogs/ [ 279.524913] Lustre: lustre-OST0000-osc-ffff9924c707b000: disconnect after 23s idle [ 286.054504] Lustre: DEBUG MARKER: oleg210-client.virtnet: executing yml_node [ 292.702389] Lustre: DEBUG MARKER: Client: 2.17.57.44 [ 296.131740] Lustre: DEBUG MARKER: MDS: 2.17.57.44 [ 300.018982] Lustre: DEBUG MARKER: OSS: 2.17.57.44 [ 302.049929] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Aug 19 21:45:26 EDT 2026 [ 320.529274] Lustre: Unmounted lustre-client [ 351.806487] Key type lgssc unregistered [ 351.997977] LNet: 8449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.004780] LNetError: 8449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 352.017346] LNet: Removed LNI 192.168.202.10@tcp [ 352.588316] Key type .llcrypt unregistered [ 352.591154] Key type ._llcrypt unregistered [ 383.112908] Key type ._llcrypt registered [ 383.115223] Key type .llcrypt registered [ 383.350661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 383.367968] alg: No test for adler32 (adler32-zlib) [ 384.300934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 384.362641] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 384.380316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 388.408428] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 388.628787] LNet: 9335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.644561] LNetError: 9335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 388.688258] LNet: Removed LNI 192.168.202.10@tcp [ 389.465595] Key type .llcrypt unregistered [ 389.473755] Key type ._llcrypt unregistered [ 395.692903] Key type ._llcrypt registered [ 395.702709] Key type .llcrypt registered [ 397.245871] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:47:01 (1787190421) [ 397.604863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 397.619199] alg: No test for adler32 (adler32-zlib) [ 398.542960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.618207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.637786] LNet: 9755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.663478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.712795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 398.736282] LNet: 9762:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 406.766280] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:47:11 (1787190431) [ 407.039216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.073264] LNet: 10341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.092900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.165784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 413.907587] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:47:18 (1787190438) [ 414.173661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.199393] LNet: 10929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.212316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.306696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 422.662290] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:47:26 (1787190446) [ 422.751549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.798220] LNet: 11508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.832052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.942927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 430.087637] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:47:34 (1787190454) [ 430.502285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.543831] LNet: 12099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.564550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.616649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 431.058717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 437.968479] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:47:42 (1787190462) [ 438.284461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.336491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.466661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 438.958354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 439.362116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 447.879744] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:47:52 (1787190472) [ 448.010649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.071617] LNet: 13281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.077521] LNet: 13281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 448.102448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.279325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 456.291555] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:48:01 (1787190481) [ 456.674707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.710986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.785782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 456.808988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 457.345303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 457.402967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 457.980706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 458.042864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 458.721671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 458.790733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 459.451498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 459.513749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 460.051165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 460.069661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 460.780719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 460.835085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 461.468687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 461.517364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 462.088159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 462.142791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 462.484330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 462.515813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 462.868401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 462.885397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 463.277631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 463.327674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 463.760952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 463.801459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 470.625121] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:48:15 (1787190495) [ 470.837461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.877698] LNet: 14551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 470.886474] LNet: 14551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 470.954436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.064661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 471.121114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 477.546267] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:48:22 (1787190502) [ 477.760598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.784629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.839322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 477.870458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 485.600456] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:48:29 (1787190509) [ 485.912627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.972479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.045937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 486.077436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 492.801197] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:48:37 (1787190517) [ 493.073953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.101745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.178438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 493.239823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 500.349478] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:48:44 (1787190524) [ 500.797617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.868714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.034483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 501.093476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 510.161401] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:48:54 (1787190534) [ 510.295855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.333203] LNet: 17482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 510.340937] LNet: 17482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 510.374918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.490260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 510.524773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 517.703580] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:49:02 (1787190542) [ 518.185247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.261949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.378139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.056733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.891289] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:49:11 (1787190551) [ 527.307956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.365684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.674240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 527.875867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.649636] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:49:19 (1787190559) [ 534.846613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.914621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.004388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 535.236699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 542.088407] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:49:26 (1787190566) [ 542.215105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.273807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.023553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 543.322504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 551.449983] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:49:36 (1787190576) [ 551.903876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.972406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.076530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.146525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 558.866601] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:49:43 (1787190583) [ 559.234478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.300570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.381844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 559.453638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 567.480965] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:49:51 (1787190591) [ 567.535791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.567539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.649423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 574.739941] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:49:59 (1787190599) [ 574.889202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.938062] LNet: 22337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.943455] LNet: 22337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 574.979280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.097724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 582.047549] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:50:06 (1787190606) [ 582.645752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.718849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.888127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 583.531360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 591.443246] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:50:16 (1787190616) [ 591.838355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.909506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.025214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 592.073980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 599.855745] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:50:24 (1787190624) [ 599.954698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.013397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.098768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 600.127043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 607.785774] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:50:31 (1787190631) [ 607.924230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.969627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.120718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 608.170930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 616.327757] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:50:40 (1787190640) [ 616.710990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.818185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.947687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 617.035476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 625.432338] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:50:49 (1787190649) [ 625.570481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.698965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.848990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 625.945362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 625.966949] LNetError: 25876:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 626.023516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 626.054927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 634.239566] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:50:58 (1787190658) [ 634.574818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.622659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.737422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E4RCcYUFum [ 634.841873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E4RCcYUFum [ 634.917249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E4RCcYUFum [ 642.596572] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:51:07 (1787190667) [ 642.702399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.768312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.868089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 642.917916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 642.991579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 649.354599] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 21:51:13 (1787190673) [ 649.412227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 649.452830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 649.516234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 649.535898] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 650.926802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 652.188958] Lustre: Lustre: Build Version: 2.17.57_44_gafaaa93 [ 654.031207] Key type lgssc registered [ 655.234424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 659.747894] LNetError: 27637:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.10@tcp99 has no available nets [ 662.809796] Key type lgssc unregistered [ 663.249066] LNetError: 28749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.269414] LNet: Removed LNI 192.168.202.10@tcp [ 664.020570] Key type .llcrypt unregistered [ 664.023668] Key type ._llcrypt unregistered [ 670.388971] Key type ._llcrypt registered [ 670.391422] Key type .llcrypt registered [ 672.086615] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:51:36 (1787190696) [ 672.421900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.435911] alg: No test for adler32 (adler32-zlib) [ 673.388499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.496896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 676.640082] LNet: Message forwarding will be enabled in 2s [ 679.711167] LNet: Message forwarding enabled [ 680.258763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 680.276381] LNet: Message forwarding will stop in 2s to 6s [ 684.829764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 685.371871] LNet: Message forwarding disabled [ 687.438028] LNet: Message forwarding will be enabled in 2s [ 690.527231] LNet: Message forwarding enabled [ 694.466679] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:51:59 (1787190719) [ 694.736219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.763628] LNet: 30156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.358694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.449555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 701.986867] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:52:06 (1787190726) [ 702.133127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.169354] LNet: 30733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.216180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.272476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 702.299719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 702.327615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 702.352674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 702.388787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 702.421436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 702.461132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 702.497890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 702.533646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 702.551783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 702.567823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 702.587573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 702.609829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 702.639484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 702.667321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 702.704652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 702.752178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 702.804697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 702.836435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 702.856894] LNetError: 30781:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 708.537811] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:52:13 (1787190733) [ 708.640524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 708.671495] LNet: 31346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 708.709548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 708.837623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.105686] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:52:20 (1787190740) [ 716.984065] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg210-server [ 718.605816] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:52:23 (1787190743) [ 718.941896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.957643] LNet: 32284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.982168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.042821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 719.068748] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 719.912561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[11-15]@tcp [ 719.931512] LNet: 32417:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 719.983733] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 720.012198] LNetError: 32288:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.11@tcp has gone from up to down [ 721.006516] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.12@tcp is being used as a gateway but routing feature is not turned on [ 721.013407] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 722.023226] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 722.033031] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 724.072674] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.12@tcp is being used as a gateway but routing feature is not turned on [ 724.083159] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 726.558312] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:52:31 (1787190751) [ 728.162782] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.11@tcp is being used as a gateway but routing feature is not turned on [ 728.182020] LNetError: 32288:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 728.748741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg210-server [ 730.503801] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:52:35 (1787190755) [ 730.679226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.735551] LNet: 33347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.743903] LNetError: 33347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.783022] LNet: Removed LNI 192.168.202.10@tcp [ 730.844295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.944180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 730.984656] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 732.029152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 732.066892] LNet: 33470:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 732.075707] LNet: 33470:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 732.098629] LNetError: 33356:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.13@tcp has gone from up to down [ 732.100942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[11-16/2]@tcp [ 732.108056] LNetError: 33356:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 738.714151] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:52:43 (1787190763) [ 738.989923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.018463] LNet: 34064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.024147] LNetError: 34064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.040112] LNet: Removed LNI 192.168.202.10@tcp [ 739.076916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.174191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 739.218147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.247941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.306156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 739.362772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.398329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.440441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 739.491423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.546446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.629427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 739.724843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.762311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.832800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 739.898364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.932534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.991887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 740.078528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.113503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.174882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 740.289785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.351761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.407613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 740.470932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.507375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.558480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 740.651252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.680538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.743801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 740.815598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.855094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.926278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 740.988756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.024822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.084350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.917362] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:52:51 (1787190771) [ 748.714579] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg210-server [ 750.238807] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:52:55 (1787190775) [ 752.319780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg210-server [ 753.913512] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:52:58 (1787190778) [ 754.714175] Key type .llcrypt unregistered [ 754.717216] Key type ._llcrypt unregistered [ 761.430270] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 761.650186] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.667921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 761.679146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.346703] Key type ._llcrypt registered [ 764.349282] Key type .llcrypt registered [ 764.585538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.599359] alg: No test for adler32 (adler32-zlib) [ 765.479169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.524485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.564441] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 766.421357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 766.464763] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.228734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 768.251596] LNet: Removed LNI 10.1.2.3@tcp [ 768.377055] LNet: 36193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.384984] LNetError: 36193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.416463] LNet: Removed LNI 192.168.202.10@tcp [ 768.875454] Key type .llcrypt unregistered [ 768.876818] Key type ._llcrypt unregistered [ 781.137159] Key type ._llcrypt registered [ 781.139442] Key type .llcrypt registered [ 782.439893] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:53:27 (1787190807) [ 783.064370] Key type .llcrypt unregistered [ 783.072711] Key type ._llcrypt unregistered [ 790.970961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.218442] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.249030] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.256071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.791257] Key type ._llcrypt registered [ 793.794215] Key type .llcrypt registered [ 794.018210] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.058549] alg: No test for adler32 (adler32-zlib) [ 794.962104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.016305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.041790] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 795.622416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 795.646396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.925166] LNet: 37695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.932851] LNetError: 37695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 797.938762] LNet: Removed LNI 10.1.2.3@tcp [ 797.948203] LNet: Removed LNI 192.168.202.10@tcp [ 798.425795] Key type .llcrypt unregistered [ 798.427794] Key type ._llcrypt unregistered [ 811.136283] Key type ._llcrypt registered [ 811.138533] Key type .llcrypt registered [ 812.426309] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:53:57 (1787190837) [ 812.953975] Key type .llcrypt unregistered [ 812.955461] Key type ._llcrypt unregistered [ 820.502077] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 820.720200] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 820.729899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 820.748606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.478155] Key type ._llcrypt registered [ 823.481089] Key type .llcrypt registered [ 823.600323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.613168] alg: No test for adler32 (adler32-zlib) [ 824.608318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.662624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 824.692647] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 826.723716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 826.758938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 828.549979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 828.596714] LNet: Removed LNI 10.1.2.3@tcp [ 828.610132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 828.648718] LNet: Removed LNI 10.1.2.31@tcp [ 828.665717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 828.686857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 829.342717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 829.366646] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 830.224494] LNet: 39624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.230632] LNetError: 39624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 830.235688] LNet: Removed LNI 10.1.2.31@tcp [ 830.248807] LNet: Removed LNI 10.1.2.3@tcp [ 830.827609] Key type .llcrypt unregistered [ 830.829929] Key type ._llcrypt unregistered [ 843.766582] Key type ._llcrypt registered [ 843.768682] Key type .llcrypt registered [ 845.168679] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:54:29 (1787190869) [ 845.743317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.752984] alg: No test for adler32 (adler32-zlib) [ 846.731115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.802841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.839347] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 847.601799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.621151] LNet: 40581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.626712] LNetError: 40581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.641194] LNet: Removed LNI 192.168.202.10@tcp [ 847.665715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.747342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.10@tcp [ 847.770446] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 848.478893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.10@tcp [ 848.500805] LNet: Removed LNI 192.168.202.10@tcp [ 848.564659] LNet: 40716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.082392] Key type .llcrypt unregistered [ 849.087707] Key type ._llcrypt unregistered [ 861.761585] Key type ._llcrypt registered [ 861.765427] Key type .llcrypt registered [ 863.496295] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:54:47 (1787190887) [ 865.872980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg210-server [ 868.064504] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:54:52 (1787190892) [ 868.561354] Key type .llcrypt unregistered [ 868.563067] Key type ._llcrypt unregistered [ 875.814664] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 875.888573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 875.896785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 875.907485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 878.412360] Key type ._llcrypt registered [ 878.414354] Key type .llcrypt registered [ 878.581696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.591971] alg: No test for adler32 (adler32-zlib) [ 879.529604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.581479] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 880.209247] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 882.084126] LNet: 42560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.090464] LNetError: 42560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.093316] LNet: Removed LNI 192.168.202.10@tcp1 [ 882.102544] LNet: Removed LNI 10.1.2.3@tcp2 [ 882.746629] Key type .llcrypt unregistered [ 882.749569] Key type ._llcrypt unregistered [ 895.479343] Key type ._llcrypt registered [ 895.481355] Key type .llcrypt registered [ 896.692132] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:55:21 (1787190921) [ 897.171775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 897.188555] alg: No test for adler32 (adler32-zlib) [ 898.142458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.188699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 898.230812] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 899.120599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.131966] LNet: 43534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.136791] LNetError: 43534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.177959] LNet: Removed LNI 192.168.202.10@tcp [ 899.196886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.248211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 899.277689] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 905.477179] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:55:30 (1787190930) [ 905.796760] LNet: 44128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.805323] LNetError: 44128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.832927] LNet: Removed LNI 192.168.202.10@tcp [ 906.403552] Key type .llcrypt unregistered [ 906.405357] Key type ._llcrypt unregistered [ 912.007868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 912.173872] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 912.177707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 912.184916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 914.769541] Key type ._llcrypt registered [ 914.771129] Key type .llcrypt registered [ 914.954351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.968777] alg: No test for adler32 (adler32-zlib) [ 915.918985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.980623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 916.001330] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 916.668498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 916.691293] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.711266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.730697] LNet: 44874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.738948] LNetError: 44874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.746144] LNet: Removed LNI 10.1.2.3@tcp [ 918.755463] LNet: Removed LNI 192.168.202.10@tcp [ 918.773281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.828207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 918.846548] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 918.848988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.394267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.419162] LNet: 44901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.426316] LNetError: 44901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.429771] LNetError: 44901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 919.444571] LNet: Removed LNI 10.1.2.3@tcp [ 919.471742] LNet: Removed LNI 192.168.202.10@tcp [ 919.509583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.587637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 919.606408] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 919.610082] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.005729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 920.038365] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 920.823342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.843850] LNet: 45063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.849248] LNetError: 45063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.854780] LNetError: 45063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 920.860202] LNet: Removed LNI 10.1.2.3@tcp [ 920.869283] LNet: Removed LNI 192.168.202.10@tcp [ 920.880856] LNet: Removed LNI 192.168.202.10@tcp2 [ 920.896702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.938655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 920.959090] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 920.964151] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.969969] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 927.479753] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:55:52 (1787190952) [ 927.880661] LNet: 45661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.889692] LNetError: 45661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.903491] LNetError: 45661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 927.912309] LNet: Removed LNI 10.1.2.3@tcp [ 927.916886] LNet: Removed LNI 192.168.202.10@tcp [ 927.932168] LNet: Removed LNI 192.168.202.10@tcp2 [ 928.594726] Key type .llcrypt unregistered [ 928.597059] Key type ._llcrypt unregistered [ 936.729111] Key type ._llcrypt registered [ 936.732779] Key type .llcrypt registered [ 937.040195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.068915] alg: No test for adler32 (adler32-zlib) [ 938.014854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.077180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 938.103187] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 938.707268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.724268] LNet: 46059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.732629] LNetError: 46059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.750630] LNet: Removed LNI 192.168.202.10@tcp [ 938.771407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.847734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 938.879965] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 939.374672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.392893] LNet: 46089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.399061] LNetError: 46089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.423695] LNet: Removed LNI 192.168.202.10@tcp [ 939.443824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.499516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 939.518992] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 940.053604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.092081] LNet: Removed LNI 192.168.202.10@tcp [ 940.120566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.184524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 940.209627] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 940.810390] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.940184] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.946489] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.952355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 943.458603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.486970] LNet: 46192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.492853] LNet: 46192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 943.501528] LNetError: 46192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.507799] LNetError: 46192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 943.524205] LNet: Removed LNI 192.168.202.10@tcp [ 943.561609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 943.624823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 943.653373] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 950.723220] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:56:15 (1787190975) [ 950.954180] LNet: 46789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 950.962546] LNetError: 46789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 950.977117] LNet: Removed LNI 192.168.202.10@tcp [ 951.789212] Key type .llcrypt unregistered [ 951.793462] Key type ._llcrypt unregistered [ 958.060275] Key type ._llcrypt registered [ 958.062639] Key type .llcrypt registered [ 958.238850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 958.251616] alg: No test for adler32 (adler32-zlib) [ 959.181882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.244606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 959.264571] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 960.066957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.080622] LNet: 47192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.084479] LNetError: 47192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.108342] LNet: Removed LNI 192.168.202.10@tcp [ 960.149375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.211884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 960.232177] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 966.532931] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:56:31 (1787190991) [ 966.722228] LNet: 47785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.727750] LNetError: 47785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.740342] LNet: Removed LNI 192.168.202.10@tcp [ 967.495151] Key type .llcrypt unregistered [ 967.497703] Key type ._llcrypt unregistered [ 974.438032] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 974.660156] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 974.665853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 974.670886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 977.056267] Key type ._llcrypt registered [ 977.057733] Key type .llcrypt registered [ 977.197828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.216091] alg: No test for adler32 (adler32-zlib) [ 978.187887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.233863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.251849] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 979.015295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.035668] LNet: 48238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.039130] LNetError: 48238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.050148] LNet: Removed LNI 192.168.202.10@tcp [ 979.062895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.110678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 979.141182] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 985.088881] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:56:50 (1787191010) [ 985.271770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.290410] LNet: 48835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.305883] LNetError: 48835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.347604] LNet: Removed LNI 192.168.202.10@tcp [ 985.378637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.468437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 990.928571] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:56:55 (1787191015) [ 991.261385] LNet: 49417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.887710] Key type .llcrypt unregistered [ 991.889288] Key type ._llcrypt unregistered [ 997.191880] Key type ._llcrypt registered [ 997.195457] Key type .llcrypt registered [ 997.391261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.408380] alg: No test for adler32 (adler32-zlib) [ 998.364286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.411990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.456328] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 999.117992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.136718] LNet: 49819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.141282] LNetError: 49819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.160291] LNet: Removed LNI 192.168.202.10@tcp [ 999.172239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.219718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 999.238468] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1004.876276] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:57:09 (1787191029) [ 1005.030542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.052460] LNet: 50412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.057394] LNetError: 50412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.074034] LNet: Removed LNI 192.168.202.10@tcp [ 1005.091966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.151612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1010.646398] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:57:15 (1787191035) [ 1012.027732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1013.413610] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:57:18 (1787191038) [ 1014.600275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1016.241854] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:57:21 (1787191041) [ 1017.904691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1019.366240] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:57:24 (1787191044) [ 1020.614513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1022.432768] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:57:27 (1787191047) [ 1023.713602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1025.421821] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:57:30 (1787191050) [ 1025.571497] LNet: 52459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.192637] Key type .llcrypt unregistered [ 1026.195705] Key type ._llcrypt unregistered [ 1032.538136] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.732292] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.736392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.745375] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1035.289845] Key type ._llcrypt registered [ 1035.292917] Key type .llcrypt registered [ 1035.510972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.530786] alg: No test for adler32 (adler32-zlib) [ 1036.489735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.567638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1036.593400] LNet: Added LNI 192.168.202.10@tcp [8/512/0/180] [ 1036.598191] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1036.599970] LNet: 52790:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1036.695908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.722796] LNet: 52809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.732815] LNetError: 52809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1036.747398] LNet: Removed LNI 10.1.2.3@tcp [ 1036.750620] LNetError: 52809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.777264] LNet: Removed LNI 192.168.202.10@tcp [ 1036.807500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.880094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1036.896122] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1036.950383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1036.967678] LNet: 52833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.972562] LNetError: 52833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.988721] LNet: Removed LNI 192.168.202.10@tcp [ 1037.007802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.056321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.077318] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1037.081077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.085635] LNet: 52841:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.170901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.207792] LNet: 52859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.218636] LNetError: 52859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.229208] LNet: Removed LNI 10.1.2.3@tcp [ 1037.232545] LNetError: 52859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.251676] LNet: Removed LNI 192.168.202.10@tcp [ 1037.286895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.352674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.389524] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1037.392560] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.395656] LNet: 52867:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1037.523110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.543896] LNet: 52893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.550100] LNetError: 52893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.557082] LNet: Removed LNI 10.1.2.3@tcp [ 1037.558641] LNetError: 52893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.576270] LNet: Removed LNI 192.168.202.10@tcp [ 1037.606321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1037.692443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1037.739648] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 1037.745515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1037.839420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1037.878579] LNet: 52921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.885190] LNetError: 52921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1037.890872] LNet: Removed LNI 10.1.2.3@tcp [ 1037.892847] LNetError: 52921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.913967] LNet: Removed LNI 192.168.202.10@tcp [ 1037.931978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1038.005896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1038.043177] LNet: Added LNI 192.168.202.10@tcp [8/256/0/182] [ 1038.046948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1038.059875] LNet: 52929:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1038.182992] LNet: Removed LNI 10.1.2.3@tcp [ 1038.192846] LNet: Removed LNI 192.168.202.10@tcp [ 1038.763357] Key type .llcrypt unregistered [ 1038.767954] Key type ._llcrypt unregistered [ 1050.394760] Key type ._llcrypt registered [ 1050.397754] Key type .llcrypt registered [ 1051.689159] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:57:56 (1787191076) [ 1053.308405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1054.939963] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:57:59 (1787191079) [ 1055.416837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1055.432898] alg: No test for adler32 (adler32-zlib) [ 1056.341823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.385619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1056.402574] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1057.101413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.10 [ 1057.120509] LNet: 54200:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1057.126901] LNet: Removed LNI 192.168.202.10@tcp [ 1063.871893] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:58:08 (1787191088) [ 1063.963730] LNet: 54764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.985781] LNetError: 54764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.016233] LNet: Removed LNI 192.168.202.10@tcp [ 1064.702182] Key type .llcrypt unregistered [ 1064.708356] Key type ._llcrypt unregistered [ 1071.988889] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1072.213557] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1075.057362] Key type ._llcrypt registered [ 1075.061748] Key type .llcrypt registered [ 1075.325605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.354837] alg: No test for adler32 (adler32-zlib) [ 1076.944613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1083.384871] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:58:28 (1787191108) [ 1083.732373] LNet: 55689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.750136] LNetError: 55689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1083.767771] LNet: Removed LNI 10.1.2.3@tcp [ 1084.493934] Key type .llcrypt unregistered [ 1084.495676] Key type ._llcrypt unregistered [ 1090.834743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1091.065544] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1091.071981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1091.076853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1093.885884] Key type ._llcrypt registered [ 1093.887451] Key type .llcrypt registered [ 1094.133784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.156271] alg: No test for adler32 (adler32-zlib) [ 1095.183273] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1103.327238] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:58:47 (1787191127) [ 1104.981864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1106.714598] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:58:51 (1787191131) [ 1106.995163] LNet: 56879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.003081] LNetError: 56879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1107.032053] LNet: Removed LNI 10.1.2.3@tcp [ 1107.686653] Key type .llcrypt unregistered [ 1107.688604] Key type ._llcrypt unregistered [ 1115.023940] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1115.377259] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1115.383447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1115.390631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1118.219604] Key type ._llcrypt registered [ 1118.226979] Key type .llcrypt registered [ 1118.471858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.505707] alg: No test for adler32 (adler32-zlib) [ 1119.505590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.614063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1128.489750] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:59:13 (1787191153) [ 1128.875536] LNet: 58113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1128.888368] LNetError: 58113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1128.896514] LNet: Removed LNI 10.1.2.3@tcp [ 1129.543435] Key type .llcrypt unregistered [ 1129.544890] Key type ._llcrypt unregistered [ 1137.049390] Key type ._llcrypt registered [ 1137.053392] Key type .llcrypt registered [ 1137.247819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1137.265796] alg: No test for adler32 (adler32-zlib) [ 1138.230783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1138.299829] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1147.039855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1147.668827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1147.708061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1147.733750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1147.751824] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1147.955248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.990219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1148.016990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1148.045146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1148.080415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1148.106584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1148.138609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1148.167669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1148.187701] LNet: 58554:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1148.201794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1148.247900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1148.288136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1148.327820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1148.410781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.510992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.553017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1149.087899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1149.125635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1149.166936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1149.182369] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1149.396981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.430281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.468957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.493236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.519280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.548872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.573932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.602035] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1149.608830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.631139] LNet: 58664:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.653944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.684169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.709110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.741699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.767205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.794824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.821861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1150.248887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1150.293528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1150.327242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1150.355347] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1150.671382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1150.713599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1150.755118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1150.803730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1150.836566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1150.862739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1150.890694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1150.921790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1150.945198] LNet: 58773:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1150.970195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1151.003282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1151.032423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1151.057348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1151.078850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1151.102784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1151.135763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1151.654787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1151.694282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1151.738924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1151.984664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.013970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1152.050145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.078241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.116761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.146890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.181700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.228655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.266875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.302087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.323731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.372803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.426266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.472262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.514245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1152.985932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1153.017147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1153.042805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1153.059984] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1153.064683] LNet: Skipped 2 previous similar messages [ 1153.267545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.292201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.321238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.349617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.372627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.405690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.432614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.460568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.482341] LNet: 58991:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.487798] LNet: 58991:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1153.502802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.536658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.575787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.629571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.676620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.721218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.777274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1154.311610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1154.344472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1154.395804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1154.658135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.688896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.720686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.757324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.793281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.832540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.876414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.909744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.938319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.970618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.993100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.020709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.070237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.108441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.151853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1155.473963] LNet: 59145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.498394] LNetError: 59145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.514406] LNet: Removed LNI 192.168.202.10@tcp [ 1156.009150] Key type .llcrypt unregistered [ 1156.012731] Key type ._llcrypt unregistered [ 1174.248278] Key type ._llcrypt registered [ 1174.252691] Key type .llcrypt registered [ 1175.515807] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:00:00 (1787191200) [ 1176.624354] Key type .llcrypt unregistered [ 1176.626485] Key type ._llcrypt unregistered [ 1183.660444] Key type ._llcrypt registered [ 1183.662599] Key type .llcrypt registered [ 1183.876410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1183.895907] alg: No test for adler32 (adler32-zlib) [ 1184.777676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1184.824663] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1193.268903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1203.595526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1203.639596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1203.671572] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1204.867698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1204.902354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1204.934237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1204.970399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1205.006292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 1 [ 1205.036472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 1 [ 1205.065133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 1 [ 1205.094500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 1 [ 1205.128546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1205.146345] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1205.151701] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1205.352803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1205.385186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1205.421729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1205.443787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1205.469892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1205.498470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1205.532611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1205.571543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1205.586769] LNet: 60616:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1205.602121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1205.630600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1205.657466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1205.682593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1205.705473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.732588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1205.762540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1206.223710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1206.253736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1206.289500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1206.319286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1206.346996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 1 [ 1206.373899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 1 [ 1206.411577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 1 [ 1206.451291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 1 [ 1206.488609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1206.504829] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1206.508576] LNet: Skipped 2 previous similar messages [ 1206.673245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1206.702837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1206.744242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1206.777171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1206.803470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1206.838075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1206.863798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1206.886226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1206.903835] LNet: 60738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1206.923686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1206.946915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1206.973321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1206.998440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1207.022456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1207.052453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1207.070597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1207.505880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1207.563441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1207.628522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1207.684784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1207.727318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 1 [ 1207.767382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 1 [ 1207.812460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 1 [ 1207.855939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 1 [ 1207.888531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1207.906025] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1207.909974] LNet: Skipped 2 previous similar messages [ 1208.082961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.112687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.139495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.167728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1208.191338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1208.215937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1208.236300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1208.260621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1208.298423] LNet: 60859:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1208.317963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1208.351314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.377632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.411721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1208.441671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.466441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.493164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1208.913775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1208.940649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1208.965597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1208.992873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1209.014664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 1 [ 1209.035914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 1 [ 1209.073317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 1 [ 1209.112147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 1 [ 1209.155990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1209.347617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.371729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1209.388724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.418461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.446682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.489324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.525609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.569439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.631190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.670734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.716666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.751448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.783477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.824776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.852302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.325934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1210.360864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1210.416622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1210.454615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1210.497732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 1 [ 1210.554812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 1 [ 1210.604443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 1 [ 1210.672783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 1 [ 1210.723450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1210.753419] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1210.758280] LNet: Skipped 5 previous similar messages [ 1210.977483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.007917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.045905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.077543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.120704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.152839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.168679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.182913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.193320] LNet: 61102:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.196277] LNet: 61102:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1211.207059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.235641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.271159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.324272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.369849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.414351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.440422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1211.884259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1211.912994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1211.943368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1211.973797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1212.000459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_error -r 1 [ 1212.030730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_error -r 1 [ 1212.060987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_error -r 1 [ 1212.093602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_error -r 1 [ 1212.129529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1212.429751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.472565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1212.499275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.541788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.567338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.592603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.627302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.656757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.687326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.715265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.746156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.769699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.792803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.823819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.862701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.749454] LNet: 61303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1218.779853] LNetError: 61303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1218.791163] LNet: Removed LNI 192.168.202.10@tcp [ 1218.818233] LNet: Removed LNI 192.168.202.10@tcp1 [ 1219.466813] Key type .llcrypt unregistered [ 1219.468368] Key type ._llcrypt unregistered [ 1237.069743] Key type ._llcrypt registered [ 1237.071160] Key type .llcrypt registered [ 1238.279460] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:01:03 (1787191263) [ 1238.920602] Key type .llcrypt unregistered [ 1238.922974] Key type ._llcrypt unregistered [ 1245.050368] Key type ._llcrypt registered [ 1245.056833] Key type .llcrypt registered [ 1245.226923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1245.247632] alg: No test for adler32 (adler32-zlib) [ 1246.134373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1246.153757] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1253.975371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1254.618663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1254.649657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1254.678893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1254.701393] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1254.933418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.967383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1255.001859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1255.030861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1255.056399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1255.087136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1255.117074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1255.144930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1255.158705] LNet: 62584:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.173593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1255.200627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1255.225955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1255.245680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.272160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.300814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.336979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1255.840275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1255.895696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1255.924657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1255.941800] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1256.145722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1256.177783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1256.205839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1256.251688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1256.293796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1256.343668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1256.367860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1256.406601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1256.428685] LNet: 62693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1256.446857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1256.484434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1256.515156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1256.544186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1256.585374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1256.628366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1256.676907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1257.208483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1257.241878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1257.280639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1257.298399] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1257.546436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.593944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.647273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.694333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.717417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.749797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.775401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.801342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.817348] LNet: 62802:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.839406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.868694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.894841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1257.922620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1257.951677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1257.986932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.022643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1258.328534] LNet: 62847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.332991] LNetError: 62847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.357565] LNet: Removed LNI 192.168.202.10@tcp [ 1258.796563] Key type .llcrypt unregistered [ 1258.802422] Key type ._llcrypt unregistered [ 1274.991919] Key type ._llcrypt registered [ 1274.993691] Key type .llcrypt registered [ 1276.195183] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:01:40 (1787191300) [ 1277.028447] Key type .llcrypt unregistered [ 1277.032045] Key type ._llcrypt unregistered [ 1282.339669] Key type ._llcrypt registered [ 1282.346764] Key type .llcrypt registered [ 1282.528637] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1282.540569] alg: No test for adler32 (adler32-zlib) [ 1283.431549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1283.462531] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1291.514036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1299.499535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1299.535731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1299.555360] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1300.630425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1300.655167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1300.684541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1300.704656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1300.740711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 1 [ 1300.793360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 1 [ 1300.850854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 1 [ 1300.896316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 1 [ 1300.928858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1300.944872] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1300.948350] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1301.173683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1301.211734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1301.248515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1301.281851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1301.307286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1301.331597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1301.366854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1301.416217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1301.433602] LNet: 64317:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1301.445898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1301.468249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1301.490756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1301.514179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1301.540438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.570637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1301.601042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1301.630581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.086669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1302.118977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1302.146809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1302.186269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1302.214839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_error -r 1 [ 1302.247787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_error -r 1 [ 1302.278715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_error -r 1 [ 1302.309545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_error -r 1 [ 1302.343748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1302.362935] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1302.373264] LNet: Skipped 2 previous similar messages [ 1302.565499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.587173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1302.610489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.632931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.660820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.685789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.708613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.738620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.754865] LNet: 64440:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.772368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.794841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.817165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.836401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.858291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.884912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.910517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1302.945690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1303.367854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1303.400610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1303.426559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1303.457503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1303.479538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_timeout -r 1 [ 1303.503644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_timeout -r 1 [ 1303.533604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_timeout -r 1 [ 1303.568294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_timeout -r 1 [ 1303.596451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.110@tcp [ 1303.612452] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1303.814285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.834954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1303.862369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1303.891984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1303.919387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1303.952277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1303.981389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.009476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.027378] LNet: 64563:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.043606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.073229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.102358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.129775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.154735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.179136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.202455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1304.225908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.497844] LNet: 64645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.514629] LNetError: 64645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.520844] LNet: Removed LNI 192.168.202.10@tcp [ 1308.538237] LNet: Removed LNI 192.168.202.10@tcp1 [ 1308.841933] Key type .llcrypt unregistered [ 1308.844773] Key type ._llcrypt unregistered [ 1321.719826] Key type ._llcrypt registered [ 1321.722991] Key type .llcrypt registered [ 1322.977115] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:02:27 (1787191347) [ 1324.281068] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1325.967794] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:02:30 (1787191350) [ 1326.741636] Key type .llcrypt unregistered [ 1326.743125] Key type ._llcrypt unregistered [ 1332.952800] Key type ._llcrypt registered [ 1332.954448] Key type .llcrypt registered [ 1333.129464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1333.140656] alg: No test for adler32 (adler32-zlib) [ 1334.074988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1334.109469] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1342.273673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1343.024334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1343.060685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1343.120043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1343.141825] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1343.328909] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1343.424576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1343.479918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1343.529806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1343.567927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1343.605489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1343.628657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1343.680937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1343.735675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1343.763996] LNet: 66191:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.789973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.831408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.889084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.956758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1344.000386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1344.041977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1344.377610] LNet: 66234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.385871] LNetError: 66234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.403623] LNet: Removed LNI 192.168.202.10@tcp [ 1344.913373] Key type .llcrypt unregistered [ 1344.915720] Key type ._llcrypt unregistered [ 1364.393196] Key type ._llcrypt registered [ 1364.397795] Key type .llcrypt registered [ 1364.538062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1364.546178] alg: No test for adler32 (adler32-zlib) [ 1365.571840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1365.594561] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1374.613644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1383.408327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.442218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.460105] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1385.131609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1385.194179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1385.222394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1385.267467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1385.295555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e network_timeout -r 1 [ 1385.343230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e network_timeout -r 1 [ 1385.412683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e network_timeout -r 1 [ 1385.473680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e network_timeout -r 1 [ 1385.515083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1385.543688] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1385.805942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1385.848829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1385.887783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1386.020760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1386.076562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1386.088637] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1386.088714] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1386.102784] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1386.121657] LNet: Skipped 1 previous similar message [ 1386.168560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1386.197133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1386.215173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1386.225154] LNet: 67117:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1386.236740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1386.256324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1386.278225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.299980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1386.335692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1386.401976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1390.484357] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1390.782516] LNet: 67195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.790480] LNetError: 67195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.795056] LNet: Removed LNI 192.168.202.10@tcp [ 1390.808450] LNet: Removed LNI 192.168.202.10@tcp1 [ 1391.145923] Key type .llcrypt unregistered [ 1391.149859] Key type ._llcrypt unregistered [ 1407.127675] Key type ._llcrypt registered [ 1407.129523] Key type .llcrypt registered [ 1408.120627] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:03:53 (1787191433) [ 1408.508752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.527398] alg: No test for adler32 (adler32-zlib) [ 1409.433504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.463236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1409.481955] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1409.946053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1409.966463] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1410.556307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1410.600436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1410.636845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1410.662766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1410.697414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1410.742650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1410.769814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1410.786641] LNetError: 68295:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1411.487391] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1411.491609] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1413.535451] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1414.224651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1417.631390] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1418.824620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1418.879996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1418.973296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.998585] LNet: 68412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.004142] LNetError: 68412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.017750] LNet: Removed LNI 192.168.202.10@tcp [ 1419.026613] LNet: Removed LNI 192.168.202.10@tcp1 [ 1419.047882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.090942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.115704] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1419.736959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1419.751216] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1420.401752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1420.437927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1420.483660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1420.514698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1420.534634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1420.574164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1420.615418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1420.662082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1420.684422] LNetError: 68684:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1421.151665] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1424.095926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1427.298033] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1427.304063] LNet: Skipped 1 previous similar message [ 1432.184883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1432.253258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1432.295651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1438.697901] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:04:23 (1787191463) [ 1438.897749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.916589] LNet: 69402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.923328] LNetError: 69402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1438.930142] LNetError: 69402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1438.938808] LNet: Removed LNI 192.168.202.10@tcp [ 1438.949756] LNet: Removed LNI 192.168.202.10@tcp1 [ 1438.968612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.022769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1439.071613] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1439.774277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1439.789090] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1440.586899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1440.668431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1440.704623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1440.740814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1440.775342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1440.807198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1440.854244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1440.994671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1441.056376] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.10@tcp [ 1441.057088] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1441.070084] LNet: Skipped 1 previous similar message [ 1456.481308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1456.611143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1456.640627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.668428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 500 [ 1456.738794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1463.309690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1463.397047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.410560] LNet: 69893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.414936] LNetError: 69893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.418438] LNetError: 69893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1463.423115] LNet: Removed LNI 192.168.202.10@tcp [ 1463.432495] LNet: Removed LNI 192.168.202.10@tcp1 [ 1463.445306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.478674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.494564] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1464.048495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1464.059588] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1464.609405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1464.641171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1464.689756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1464.710941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1464.747377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1464.791076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1464.820072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1464.857272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.10@tcp --health 0 [ 1465.505261] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1465.516961] LNet: Skipped 3 previous similar messages [ 1466.219593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1476.837509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1476.918213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1476.978487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1484.249519] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:05:08 (1787191508) [ 1484.844727] LNet: 70941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1484.850257] LNetError: 70941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1484.853738] LNetError: 70941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1484.857354] LNet: Removed LNI 192.168.202.10@tcp [ 1484.864110] LNet: Removed LNI 192.168.202.10@tcp1 [ 1485.447973] Key type .llcrypt unregistered [ 1485.453434] Key type ._llcrypt unregistered [ 1491.133931] Key type ._llcrypt registered [ 1491.136369] Key type .llcrypt registered [ 1491.394251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1491.418760] alg: No test for adler32 (adler32-zlib) [ 1492.345943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1492.392901] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1500.650358] Lustre: DEBUG MARKER: Initial discovery [ 1500.938273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1503.216536] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1503.369143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1503.394572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1503.411834] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1503.434512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1503.468829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -l 3 [ 1503.498956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.110@tcp -r 1 -m GET -e local_error [ 1503.526619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.110@tcp -r 1 -m PUT -l 6 [ 1503.649428] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1504.728254] Lustre: DEBUG MARKER: Force oleg210-client.virtnet to discover 192.168.202.110@tcp (in background) [ 1504.942919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.110@tcp [ 1506.263303] Lustre: DEBUG MARKER: Force oleg210-server to discover 192.168.202.10@tcp [ 1508.006468] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1509.346956] Lustre: DEBUG MARKER: Wait for 71709 [ 1510.826606] Lustre: DEBUG MARKER: Finished wait on 71709 [ 1511.149628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1511.202801] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1511.217887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1511.456377] LNet: 72161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1511.467603] LNetError: 72161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1511.483259] LNet: Removed LNI 192.168.202.10@tcp [ 1511.882874] Key type .llcrypt unregistered [ 1511.887678] Key type ._llcrypt unregistered [ 1527.925792] Key type ._llcrypt registered [ 1527.928556] Key type .llcrypt registered [ 1529.195839] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:05:53 (1787191553) [ 1529.915286] Key type .llcrypt unregistered [ 1529.919764] Key type ._llcrypt unregistered [ 1537.687550] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1537.904241] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1537.919581] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1537.932966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1540.618526] Key type ._llcrypt registered [ 1540.623246] Key type .llcrypt registered [ 1540.784773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1540.799189] alg: No test for adler32 (adler32-zlib) [ 1541.917248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.028538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.062643] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1542.928153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1542.947240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1552.502929] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:06:17 (1787191577) [ 1552.930701] LNet: 74241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.944449] LNetError: 74241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1552.949070] LNet: Removed LNI 10.1.2.3@tcp [ 1552.966222] LNet: Removed LNI 192.168.202.10@tcp [ 1553.739125] Key type .llcrypt unregistered [ 1553.746980] Key type ._llcrypt unregistered [ 1560.816969] Key type ._llcrypt registered [ 1560.819379] Key type .llcrypt registered [ 1561.000334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1561.034415] alg: No test for adler32 (adler32-zlib) [ 1562.055272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.144626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1562.198334] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1563.112152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.145224] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1564.287777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.10@tcp [ 1565.343131] LNetError: Unexpected error -22 connecting to 192.168.202.10@tcp at host 192.168.202.10:988 [ 1571.293365] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:06:35 (1787191595) [ 1571.423649] LNet: 75368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1571.448376] LNetError: 75368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1571.453439] LNet: Removed LNI 10.1.2.3@tcp [ 1571.482670] LNet: Removed LNI 192.168.202.10@tcp [ 1572.370616] Key type .llcrypt unregistered [ 1572.373370] Key type ._llcrypt unregistered [ 1579.202996] Key type ._llcrypt registered [ 1579.211228] Key type .llcrypt registered [ 1579.564114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1579.593966] alg: No test for adler32 (adler32-zlib) [ 1580.500331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.549824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1580.586451] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1581.230548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1581.250943] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 1582.110495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.10@tcp1 --nid 192.168.202.10@tcp2 [ 1586.030824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp2 [ 1586.056731] LNetError: 76217:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.10@tcp1 to 192.168.202.10@tcp2 - Route table may be misconfigured [ 1586.080106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp2 192.168.202.10@tcp1 [ 1593.041978] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:06:57 (1787191617) [ 1593.137032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.158950] LNet: 76784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.179623] LNetError: 76784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.209767] LNet: Removed LNI 192.168.202.10@tcp1 [ 1594.213823] LNetError: 76784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1595.239983] LNet: Removed LNI 192.168.202.10@tcp2 [ 1595.271133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.310205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1595.335370] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1595.955337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1595.981303] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1596.623190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 1596.654398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.10@tcp [ 1596.694776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp1 192.168.202.10@tcp1 [ 1596.733814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp -e network_timeout [ 1596.772236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -e network_timeout [ 1596.812128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -e network_timeout [ 1596.844663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -e network_timeout [ 1596.884510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1596.919505] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1596.937132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1597.043113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1597.144908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1597.343910] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1602.592752] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:07:07 (1787191627) [ 1602.804872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.819766] LNet: 77651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.823632] LNetError: 77651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.827328] LNet: Removed LNI 192.168.202.10@tcp [ 1603.879447] LNet: Removed LNI 192.168.202.10@tcp1 [ 1603.915187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1604.015361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1604.268147] LNet: 77692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1604.552759] Key type .llcrypt unregistered [ 1604.556630] Key type ._llcrypt unregistered [ 1615.568490] Key type ._llcrypt registered [ 1615.570949] Key type .llcrypt registered [ 1617.048109] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:07:21 (1787191641) [ 1617.489971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1617.509430] alg: No test for adler32 (adler32-zlib) [ 1618.459439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.095989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1622.067295] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:07:26 (1787191646) [ 1622.177820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.215736] LNet: 78804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.253794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.355966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.386452] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1623.094810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1623.117310] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1624.305816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1624.352623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp1 [ 1624.392392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp1 [ 1630.719298] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:07:35 (1787191655) [ 1632.227350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg210-server [ 1633.779754] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:07:38 (1787191658) [ 1635.264863] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg210-server [ 1636.767835] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:07:41 (1787191661) [ 1638.144394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg210-server [ 1639.889572] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:07:44 (1787191664) [ 1641.737588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg210-server [ 1643.147453] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:07:48 (1787191668) [ 1645.118388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg210-server [ 1646.992580] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:07:51 (1787191671) [ 1648.743291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg210-server [ 1650.191548] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:07:55 (1787191675) [ 1651.568807] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg210-server [ 1653.559069] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:07:58 (1787191678) [ 1655.337708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg210-server [ 1657.002704] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:08:01 (1787191681) [ 1658.964463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg210-server [ 1660.567670] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:08:05 (1787191685) [ 1660.890502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.924115] LNet: 82683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.932171] LNetError: 82683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.940305] LNet: Removed LNI 192.168.202.10@tcp [ 1661.990491] LNet: Removed LNI 192.168.202.10@tcp1 [ 1662.059942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.159645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1662.184696] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1662.965634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1663.115476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1663.328934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.351522] LNet: 82831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.359888] LNetError: 82831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.363164] LNetError: 82831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1663.403865] LNet: Removed LNI 192.168.202.10@tcp [ 1663.465322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.576386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1663.629101] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1664.348614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1664.489758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.509550] LNetError: 82976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.574698] LNet: Removed LNI 192.168.202.10@tcp [ 1665.625768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.719763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.743908] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1666.537906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1666.656287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1666.918355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.944934] LNet: 83123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.951829] LNet: 83123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1666.962408] LNetError: 83123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.975796] LNet: Removed LNI 192.168.202.10@tcp [ 1667.036811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.154375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1667.210602] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1668.005418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1668.244370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.353311] LNet: Removed LNI 192.168.202.10@tcp [ 1669.412504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.542538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.584094] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1670.555469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1670.649780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1670.849922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1671.910317] LNet: Removed LNI 192.168.202.10@tcp [ 1671.944993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.074807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1672.116908] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1673.059408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1673.346973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1673.370840] LNet: 83561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.376189] LNet: 83561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1673.388647] LNetError: 83561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.398516] LNetError: 83561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1673.430151] LNet: Removed LNI 192.168.202.10@tcp [ 1673.496567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.576201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1673.620860] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1674.344645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1674.402417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1674.549538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.623261] LNet: Removed LNI 192.168.202.10@tcp [ 1675.669545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.737394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1675.776098] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1676.383955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1676.644593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.735679] LNet: Removed LNI 192.168.202.10@tcp [ 1677.756381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.825505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.867751] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1678.618840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1678.677746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1678.972049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.041398] LNet: Removed LNI 192.168.202.10@tcp [ 1680.067683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.114732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1680.137547] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1680.739588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1680.919865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.960940] LNet: Removed LNI 192.168.202.10@tcp [ 1681.984515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.033130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1682.065694] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1682.592519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1682.658456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1684.148325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.183397] LNet: 84302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.192546] LNet: 84302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1684.198477] LNetError: 84302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.201932] LNetError: 84302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1685.222823] LNet: Removed LNI 192.168.202.10@tcp [ 1685.252951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.325773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1685.357492] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1686.241760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1686.561964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.654377] LNet: Removed LNI 192.168.202.10@tcp [ 1687.722145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.874808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.937976] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1688.802658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1688.932437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.000459] LNet: Removed LNI 192.168.202.10@tcp [ 1689.025368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.121876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.161765] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1690.032658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1690.105035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.129986] LNet: Removed LNI 192.168.202.10@tcp [ 1690.146374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.185939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1690.259422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.337286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.414889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1690.484618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.515840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.556413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1690.630264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.674915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.766560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1690.798966] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1691.536541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.558981] LNet: Removed LNI 192.168.202.10@tcp [ 1691.577420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.626028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1697.819096] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:08:42 (1787191722) [ 1697.885321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.945704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.005063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1698.046491] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1699.224517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.253802] LNet: Removed LNI 192.168.202.10@tcp231 [ 1699.271510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.336991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1699.361324] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1699.366961] LNet: 85615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1699.700501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1699.728813] LNet: Removed LNI 192.168.202.10@tcp231 [ 1699.752861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1699.779582] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1700.579493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.594626] LNet: 85759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1700.598231] LNet: 85759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1700.603452] LNetError: 85759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.606598] LNetError: 85759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1700.622774] LNet: Removed LNI 192.168.202.10@tcp231 [ 1700.635748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.695765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1700.718706] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 1700.724341] LNet: 85767:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1700.996433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1701.025960] LNet: Removed LNI 192.168.202.10@tcp231 [ 1701.036841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1701.057700] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/0] [ 1701.924883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.979899] LNet: Removed LNI 192.168.202.10@tcp231 [ 1702.022169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.183121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1702.234766] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 1702.242920] LNet: 85918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1703.054474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1703.086910] LNet: Removed LNI 192.168.202.10@tcp231 [ 1703.104442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1703.174124] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/60] [ 1704.309314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.335476] LNet: Removed LNI 192.168.202.10@tcp231 [ 1704.362137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.432599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1704.456729] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1704.461423] LNet: 86070:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1704.859183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1704.890955] LNet: Removed LNI 192.168.202.10@tcp231 [ 1704.914648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1704.939578] LNet: Added LNI 192.168.202.10@tcp231 [8/256/0/180] [ 1712.003916] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:08:56 (1787191736) [ 1712.124615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.167342] LNet: Removed LNI 192.168.202.10@tcp231 [ 1712.201968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.246971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1712.264583] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1713.140344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1713.174125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1713.217181] LNet: Removed LNI 192.168.202.10@tcp232 [ 1713.227458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1713.250649] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1713.990456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1714.030564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1714.079592] LNet: Removed LNI 192.168.202.10@tcp232 [ 1714.101757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1714.136484] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1714.789479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1714.815754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1714.845176] LNet: Removed LNI 192.168.202.10@tcp232 [ 1714.861474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1714.885368] LNet: Added LNI 192.168.202.10@tcp232 [8/256/0/180] [ 1715.534454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1715.560675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1715.588242] LNet: Removed LNI 192.168.202.10@tcp232 [ 1715.601635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1722.682194] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:09:07 (1787191747) [ 1723.700706] Key type .llcrypt unregistered [ 1723.707770] Key type ._llcrypt unregistered [ 1732.795713] Key type ._llcrypt registered [ 1732.797647] Key type .llcrypt registered [ 1732.914291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1732.925227] alg: No test for adler32 (adler32-zlib) [ 1733.828994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1733.853935] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1745.254622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1758.473157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.529695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1758.558364] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1760.149342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 1760.170851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 1760.200307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 1760.225782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 1760.264383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_interrupt -r 2 [ 1760.294869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_interrupt -r 2 [ 1760.324178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_interrupt -r 2 [ 1760.345131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_interrupt -r 2 [ 1760.369873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1760.385400] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1760.391361] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1760.609182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1760.657274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1760.703486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1760.728168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1760.750850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1760.783958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1760.814693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1760.837941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1760.850124] LNet: 88575:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1760.865503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1760.908758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1760.981893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1761.031964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1761.090904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1761.144756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1761.644169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 1761.689266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 1761.750166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 1761.802333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 1761.854606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_dropped -r 2 [ 1761.895739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_dropped -r 2 [ 1761.923156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_dropped -r 2 [ 1761.957317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_dropped -r 2 [ 1761.992060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1762.018198] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1762.315096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1762.349312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1762.389682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.441156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.465362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.493659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.527543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.558758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.590083] LNet: 88695:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1762.608682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.656405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.700772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.746030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.791964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.836709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.303622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 1763.339438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 1763.375899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 1763.413708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 1763.453918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_aborted -r 2 [ 1763.497871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_aborted -r 2 [ 1763.552828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_aborted -r 2 [ 1763.613611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_aborted -r 2 [ 1763.656768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1763.677533] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1763.914987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1763.950335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1763.985396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1764.010492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1764.036958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1764.064401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1764.096827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1764.125890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1764.141570] LNet: 88815:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.165825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1764.197981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1764.228258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1764.263618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1764.291446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1764.325079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1764.873126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 1764.917550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 1764.968625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 1765.012351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 1765.080832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_no_route -r 2 [ 1765.131442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_no_route -r 2 [ 1765.168788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_no_route -r 2 [ 1765.201834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_no_route -r 2 [ 1765.240746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1765.528760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1765.562571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1765.593642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1765.618805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1765.659845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1765.716542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1765.774806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1765.805203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1765.839741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1765.874368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1765.922733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1765.946694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1765.973266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1766.002217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.586510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 1766.640203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 1766.697708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 1766.738839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 1766.801778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e local_timeout -r 2 [ 1766.838408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e local_timeout -r 2 [ 1766.870424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e local_timeout -r 2 [ 1766.904129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e local_timeout -r 2 [ 1766.984855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1767.046773] LNet: There was an unexpected network error while writing to 192.168.202.110: rc = -22 [ 1767.052774] LNet: Skipped 1 previous similar message [ 1767.435203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.482656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1767.561223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1767.633117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1767.659633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1767.690294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1767.721547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.746862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1767.760413] LNet: 89056:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.765434] LNet: 89056:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1767.787852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1767.834899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1767.877876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1767.932481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1768.001530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.064603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.767729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 1768.819842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 1768.866461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 1768.939692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 1768.979978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp -m GET -e remote_dropped -r 2 [ 1769.047796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.110@tcp1 -m GET -e remote_dropped -r 2 [ 1769.115865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp -m GET -e remote_dropped -r 2 [ 1769.156902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.10@tcp1 -d 192.168.202.10@tcp1 -m GET -e remote_dropped -r 2 [ 1769.215127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.10@tcp 192.168.202.110@tcp [ 1769.273196] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 1769.666743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.751084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.815867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.888378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.926913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.985832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1770.028596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1770.071319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1770.110742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1770.151440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1770.199313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1770.259970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1770.306772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1770.358538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1776.298050] LNet: 89254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.303539] LNetError: 89254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.308575] LNet: Removed LNI 192.168.202.10@tcp [ 1776.318647] LNet: Removed LNI 192.168.202.10@tcp1 [ 1776.661597] Key type .llcrypt unregistered [ 1776.664683] Key type ._llcrypt unregistered [ 1793.711966] Key type ._llcrypt registered [ 1793.714587] Key type .llcrypt registered [ 1795.116454] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:10:19 (1787191819) [ 1795.513727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.526361] alg: No test for adler32 (adler32-zlib) [ 1796.419656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.466284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.481426] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1797.306566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1797.364799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.10@tcp [ 1797.390661] LNetError: 90232:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1797.471379] LNet: 1 local NIs in recovery (showing 1): 192.168.202.10@tcp [ 1797.471824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.10@tcp [ 1797.561473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1804.428559] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:10:28 (1787191828) [ 1804.496326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.529370] LNet: 90815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.534207] LNetError: 90815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.554433] LNet: Removed LNI 192.168.202.10@tcp [ 1804.588932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.669393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1804.691612] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1805.462286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1805.477224] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 1806.221489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.10@tcp [ 1806.237963] LNetError: 91089:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.10@tcp to recovery [ 1806.692438] LNet: There was an unexpected network error while writing to 192.168.202.10: rc = -22 [ 1807.467709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.10@tcp [ 1816.316393] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:10:40 (1787191840) [ 1818.377974] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg210-server [ 1820.005979] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:10:44 (1787191844) [ 1821.707310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg210-server [ 1823.425303] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:10:48 (1787191848) [ 1825.473360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg210-server [ 1826.970795] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:10:51 (1787191851) [ 1827.196414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.221542] LNet: 92720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1827.228489] LNetError: 92720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1827.237436] LNet: Removed LNI 192.168.202.10@tcp [ 1827.246363] LNet: Removed LNI 192.168.202.10@tcp1 [ 1827.259231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.299605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.315582] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1827.927439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1827.969585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1828.003297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1828.056477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1828.098531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1828.127910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1834.450783] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:10:59 (1787191859) [ 1834.709850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.752826] LNet: 93446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.765508] LNetError: 93446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.776750] LNetError: 93446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1834.797404] LNet: Removed LNI 192.168.202.10@tcp [ 1834.814726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.861417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.877910] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1841.701783] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:11:06 (1787191866) [ 1841.818864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.858856] LNet: 94145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.871683] LNetError: 94145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.894764] LNet: Removed LNI 192.168.202.10@tcp [ 1841.921908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.008922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1842.049269] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1848.482791] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:11:13 (1787191873) [ 1850.634749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1852.235416] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:11:17 (1787191877) [ 1852.959884] LNet: 95184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.970093] LNetError: 95184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.999729] LNet: Removed LNI 192.168.202.10@tcp [ 1853.761170] Key type .llcrypt unregistered [ 1853.764621] Key type ._llcrypt unregistered [ 1860.187754] Key type ._llcrypt registered [ 1860.191373] Key type .llcrypt registered [ 1860.320418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1860.332847] alg: No test for adler32 (adler32-zlib) [ 1861.248127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1861.274965] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1868.854275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 1873.986915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.110@tcp [ 1874.435224] LNet: 95609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.440671] LNetError: 95609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.457661] LNet: Removed LNI 192.168.202.10@tcp [ 1874.850399] Key type .llcrypt unregistered [ 1874.856197] Key type ._llcrypt unregistered [ 1884.224937] Key type ._llcrypt registered [ 1884.226280] Key type .llcrypt registered [ 1885.060560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1886.627257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1887.934436] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:11:52 (1787191912) [ 1889.411837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1891.328715] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:11:55 (1787191915) [ 1893.400219] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg210-server [ 1895.155392] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:11:59 (1787191919) [ 1896.505765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg210-server [ 1898.059562] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:12:03 (1787191923) [ 1899.218213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.270063] alg: No test for adler32 (adler32-zlib) [ 1900.328983] Lustre: Lustre: Build Version: 2.17.57_44_gafaaa93 [ 1900.513269] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1902.207543] Key type lgssc registered [ 1903.165630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1923.517714] Key type lgssc unregistered [ 1923.861732] LNet: 98787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.866355] LNetError: 98787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.897705] LNet: Removed LNI 192.168.202.10@tcp [ 1925.007550] Key type .llcrypt unregistered [ 1925.013977] Key type ._llcrypt unregistered [ 1940.966432] Key type ._llcrypt registered [ 1940.968533] Key type .llcrypt registered [ 1942.266524] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:12:46 (1787191966) [ 1942.772982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.788256] alg: No test for adler32 (adler32-zlib) [ 1943.812235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.899594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1943.962969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1944.034766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1944.113822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1950.572713] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:12:55 (1787191975) [ 1950.820181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.863302] LNet: 100202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1950.900976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.977237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1951.016380] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1951.987349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.004439] LNet: 100347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.013509] LNetError: 100347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.033478] LNet: Removed LNI 192.168.202.10@tcp [ 1952.053621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.113663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1952.138476] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1952.687616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.701894] LNetError: 100375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.713708] LNet: Removed LNI 192.168.202.10@tcp [ 1952.727922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.778767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1952.825946] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1959.959603] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:13:04 (1787191984) [ 1960.133838] LNet: 100965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.136359] LNet: 100965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1960.139371] LNetError: 100965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.151411] LNet: Removed LNI 192.168.202.10@tcp [ 1960.729934] Key type .llcrypt unregistered [ 1960.733322] Key type ._llcrypt unregistered [ 1967.737455] Key type ._llcrypt registered [ 1967.739109] Key type .llcrypt registered [ 1967.987189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1968.002782] alg: No test for adler32 (adler32-zlib) [ 1968.987385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.127004] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 1969.131479] LNet: *** cfs_fail_loc=e003, val=100*** [ 1970.035667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 1970.057948] LNet: *** cfs_fail_loc=e003, val=100*** [ 1970.060938] LNet: Skipped 2 previous similar messages [ 1970.068579] Ping reply received with mapping: [ 1970.071407] found metadata reply NI[192.168.202.10@tcp] [ 1970.074545] 0a ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1970.077573] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1970.080545] 5a 5a 5a 5a 5a 5a 5a 5a [ 1970.200756] LNet: 101367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1970.211174] LNetError: 101367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1970.220855] LNet: Removed LNI 192.168.202.10@tcp [ 1971.012310] Key type .llcrypt unregistered [ 1971.016552] Key type ._llcrypt unregistered [ 1982.675275] Key type ._llcrypt registered [ 1982.680723] Key type .llcrypt registered [ 1983.908830] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:13:28 (1787192008) [ 1984.569570] Key type .llcrypt unregistered [ 1984.572258] Key type ._llcrypt unregistered [ 1992.326748] Key type ._llcrypt registered [ 1992.329342] Key type .llcrypt registered [ 1992.680801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1992.726273] alg: No test for adler32 (adler32-zlib) [ 1993.856643] LNetError: 102424:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1994.399667] Key type .llcrypt unregistered [ 1994.404237] Key type ._llcrypt unregistered [ 1999.425136] Key type ._llcrypt registered [ 1999.427164] Key type .llcrypt registered [ 2000.619410] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:13:45 (1787192025) [ 2001.368627] Key type .llcrypt unregistered [ 2001.373135] Key type ._llcrypt unregistered [ 2008.335799] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2008.342148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2010.929418] Key type ._llcrypt registered [ 2010.931331] Key type .llcrypt registered [ 2011.084066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2011.110394] alg: No test for adler32 (adler32-zlib) [ 2012.077268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.148047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2022.661798] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:14:07 (1787192047) [ 2022.947577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.978838] LNet: 104212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.988574] LNetError: 104212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2023.007441] LNet: Removed LNI 10.1.2.3@tcp [ 2023.049829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.659470] Key type .llcrypt unregistered [ 2023.661982] Key type ._llcrypt unregistered [ 2031.164786] Key type ._llcrypt registered [ 2031.167762] Key type .llcrypt registered [ 2031.334660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2031.343724] alg: No test for adler32 (adler32-zlib) [ 2032.287984] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2032.385749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2032.536507] LNet: 104504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.543874] LNetError: 104504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.560558] LNet: Removed LNI 192.168.202.10@tcp [ 2033.420667] Key type .llcrypt unregistered [ 2033.422975] Key type ._llcrypt unregistered [ 2045.377801] Key type ._llcrypt registered [ 2045.380627] Key type .llcrypt registered [ 2046.760192] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:14:31 (1787192071) [ 2047.563446] Key type .llcrypt unregistered [ 2047.565038] Key type ._llcrypt unregistered [ 2053.848930] Key type ._llcrypt registered [ 2053.853704] Key type .llcrypt registered [ 2054.083815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2054.095522] alg: No test for adler32 (adler32-zlib) [ 2061.922071] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:14:46 (1787192086) [ 2062.246706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2062.297836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.372987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2062.411113] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2063.272483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2063.308812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2063.338622] LNet: Removed LNI 192.168.202.10@tcp [ 2063.579327] LNet: 106260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.876584] Key type .llcrypt unregistered [ 2063.878310] Key type ._llcrypt unregistered [ 2076.519385] Key type ._llcrypt registered [ 2076.521466] Key type .llcrypt registered [ 2078.279471] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:15:02 (1787192102) [ 2078.665317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2078.681832] alg: No test for adler32 (adler32-zlib) [ 2079.628778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.671650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.689237] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2080.301857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2080.332406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.10@tcp [ 2080.344356] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2080.347631] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2080.349657] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2080, 0, 0 [ 2080.353944] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2080.356877] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2080.359651] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2080.362558] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2080, 0, 0 [ 2080.366013] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2080.372066] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2080.374949] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2080.377968] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2080, 0, 2130 [ 2080.381799] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2080.383803] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2080.387226] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2080.389293] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2080, 0, 2130 [ 2080.392903] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2086.174579] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:15:11 (1787192111) [ 2086.612330] LNet: 107799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.616383] LNetError: 107799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.661993] LNet: Removed LNI 192.168.202.10@tcp [ 2088.515573] Key type .llcrypt unregistered [ 2088.518512] Key type ._llcrypt unregistered [ 2096.454176] Key type ._llcrypt registered [ 2096.457376] Key type .llcrypt registered [ 2096.850593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.900539] alg: No test for adler32 (adler32-zlib) [ 2097.820796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2097.849327] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2106.348907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2118.065266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.140112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2118.199887] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2119.787351] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2119.973474] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2119.976181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2119.979259] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2122.406982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2122.439548] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2133.589619] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:15:58 (1787192158) [ 2133.914989] LNet: 109306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.926483] LNetError: 109306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.931904] LNet: Removed LNI 192.168.202.10@tcp [ 2134.943226] LNet: Removed LNI 192.168.202.10@tcp1 [ 2134.950527] LNetError: 109306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2134.959490] LNetError: 109306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2135.006538] LNet: Removed LNI 10.1.2.3@tcp99 [ 2135.633670] Key type .llcrypt unregistered [ 2135.635601] Key type ._llcrypt unregistered [ 2142.725554] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2142.744124] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2145.333256] Key type ._llcrypt registered [ 2145.340382] Key type .llcrypt registered [ 2145.658889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2145.671661] alg: No test for adler32 (adler32-zlib) [ 2146.619964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.683231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.705385] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2147.581371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2147.601830] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2150.589609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2150.650577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2150.699175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2150.733698] LNetError: 110056:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2150.791387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2150.839313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2151.465340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2151.599594] LNet: 110076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.608413] LNetError: 110076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2151.616171] LNet: Removed LNI 10.1.2.3@tcp [ 2152.678367] LNet: Removed LNI 192.168.202.10@tcp [ 2153.553554] Key type .llcrypt unregistered [ 2153.556874] Key type ._llcrypt unregistered [ 2168.450763] Key type ._llcrypt registered [ 2168.453725] Key type .llcrypt registered [ 2169.993664] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:16:34 (1787192194) [ 2170.515440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2170.526725] alg: No test for adler32 (adler32-zlib) [ 2171.483911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.534642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2171.560198] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2172.488251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2172.566286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg210-client@tcp [ 2179.603540] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:16:44 (1787192204) [ 2179.751140] LNet: 111607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2179.771682] LNetError: 111607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.778376] LNet: Removed LNI 192.168.202.10@tcp [ 2181.522256] Key type .llcrypt unregistered [ 2181.527767] Key type ._llcrypt unregistered [ 2189.268526] Key type ._llcrypt registered [ 2189.270705] Key type .llcrypt registered [ 2189.541439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2189.559670] alg: No test for adler32 (adler32-zlib) [ 2190.551166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2190.589098] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2190.634076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2197.620735] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:17:02 (1787192222) [ 2199.211343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2201.240966] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:17:05 (1787192225) [ 2203.027242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2204.984605] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:17:09 (1787192229) [ 2205.243930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.268373] LNet: 113040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.277738] LNetError: 113040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.313177] LNet: Removed LNI 192.168.202.10@tcp [ 2205.334912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.382787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.412145] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2206.137268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2206.202582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2214.166184] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:17:18 (1787192238) [ 2214.406798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.428606] LNet: 113750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.441252] LNetError: 113750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.459919] LNet: Removed LNI 192.168.202.10@tcp [ 2214.482676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.576341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2214.612374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2215.387299] Key type .llcrypt unregistered [ 2215.389441] Key type ._llcrypt unregistered [ 2228.599276] Key type ._llcrypt registered [ 2228.602025] Key type .llcrypt registered [ 2230.164526] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:17:34 (1787192254) [ 2230.803094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2230.819746] alg: No test for adler32 (adler32-zlib) [ 2231.848968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.923258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2231.958497] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2232.541754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2232.600177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2239.201103] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:17:44 (1787192264) [ 2239.327510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.357473] LNet: 115309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.362885] LNetError: 115309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.393673] LNet: Removed LNI 192.168.202.10@tcp [ 2239.423413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.509765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2239.538636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2246.799686] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:17:50 (1787192270) [ 2246.922724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.960968] LNet: 115884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2246.989233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.084690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2247.843621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2247.913567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2247.992881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2248.477544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2248.541322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2248.670782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.731957] LNet: 115926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.782213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.856766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2249.465448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2249.497336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2249.567603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2250.143578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2250.169954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2256.381449] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:18:01 (1787192281) [ 2256.750729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.787390] LNet: 116526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.834282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.967701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2257.013375] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2257.715820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.749816] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2258.765976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2258.836495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2259.157723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.184912] LNetError: 116805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.195505] LNet: Removed LNI 192.168.202.10@tcp [ 2259.209208] LNet: Removed LNI 192.168.202.10@tcp2 [ 2259.251226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.350833] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2259.362485] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2259.368212] LNet: 116811:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2259.957953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2260.150567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.215169] LNet: Removed LNI 192.168.202.10@tcp [ 2261.220349] LNetError: 116849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2261.231438] LNetError: 116849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2261.244306] LNet: Removed LNI 192.168.202.10@tcp2 [ 2261.287258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.343862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2261.387689] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2262.170853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2262.202113] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2263.013961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2263.053777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2263.125713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2263.416922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.434829] LNet: 117133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2263.438857] LNet: 117133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2263.443824] LNetError: 117133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2263.449220] LNet: Removed LNI 192.168.202.10@tcp [ 2264.487257] LNet: Removed LNI 192.168.202.10@tcp2 [ 2264.527531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.608731] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2264.618250] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2264.626797] LNet: 117139:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2265.120634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2265.413791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.443589] LNet: Removed LNI 192.168.202.10@tcp [ 2266.469736] LNet: Removed LNI 192.168.202.10@tcp2 [ 2266.512599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.596641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.617530] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2267.599681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2267.641915] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2268.571887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2268.622299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2268.848627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.867826] LNetError: 117456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.873344] LNetError: 117456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2268.878497] LNet: Removed LNI 192.168.202.10@tcp [ 2268.888586] LNet: Removed LNI 192.168.202.10@tcp2 [ 2268.949416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.045208] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2269.053297] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2269.061382] LNet: 117462:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.812444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2270.030956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.135378] LNet: Removed LNI 192.168.202.10@tcp [ 2271.148941] LNet: Removed LNI 192.168.202.10@tcp2 [ 2271.185952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.252331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.274405] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2271.896901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2271.911696] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2272.687956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2272.726587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2272.786901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2273.062542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.082703] LNet: 117783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2273.086846] LNet: 117783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2274.143142] LNet: Removed LNI 192.168.202.10@tcp [ 2275.174411] LNet: Removed LNI 192.168.202.10@tcp2 [ 2275.193915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.236963] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2275.241657] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2275.245968] LNet: 117790:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2275.725567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2276.075854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.106490] LNet: Removed LNI 192.168.202.10@tcp [ 2277.161396] LNet: Removed LNI 192.168.202.10@tcp2 [ 2277.210352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.256738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.281296] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2277.928833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2277.943542] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2278.711993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2278.739306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2278.797563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2279.129277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.139152] LNetError: 118123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.141902] LNetError: 118123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2279.144679] LNet: Removed LNI 192.168.202.10@tcp [ 2279.151479] LNet: Removed LNI 192.168.202.10@tcp2 [ 2279.165225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2279.190795] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2279.193450] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2279.408659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2279.735563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2280.800365] LNet: Removed LNI 192.168.202.10@tcp [ 2280.815976] LNet: Removed LNI 192.168.202.10@tcp2 [ 2280.899785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2280.974983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2281.009300] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2281.966744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2282.005082] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2282.852141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2282.915390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2282.960942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2283.036299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2283.379134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2284.449493] LNet: Removed LNI 192.168.202.10@tcp [ 2285.482903] LNet: Removed LNI 192.168.202.10@tcp2 [ 2285.537337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.631184] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2285.648038] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2285.654045] LNet: 118489:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.655971] LNet: 118489:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2285.959208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2294.264073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2296.315678] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:18:40 (1787192320) [ 2296.507056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.537952] LNet: 119246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.552502] LNet: 119246:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2296.565566] LNetError: 119246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.570655] LNetError: 119246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2296.609672] LNet: Removed LNI 192.168.202.10@tcp [ 2297.702551] LNet: Removed LNI 192.168.202.10@tcp2 [ 2297.749946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2298.173874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2298.222028] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2299.090850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2299.106097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2299.989962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.008780] LNet: Removed LNI 10.1.2.3@tcp [ 2300.022731] LNet: Removed LNI 192.168.202.10@tcp [ 2300.046316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2300.103458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2300.127680] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2300.130918] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2300.545872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2300.562225] LNet: Removed LNI 192.168.202.10@tcp [ 2300.571767] LNet: Removed LNI 10.1.2.3@tcp [ 2306.739825] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:18:51 (1787192331) [ 2306.863992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2306.917991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.976988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2306.995842] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2307.698322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2307.737901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.767702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.796783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.820496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.847363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.889469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.921577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.954735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2307.991220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2308.023884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.10@tcp [ 2308.253542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2315.235661] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:18:59 (1787192339) [ 2315.572971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2316.652378] LNet: Removed LNI 192.168.202.10@tcp [ 2316.736603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.895284] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2316.946262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2316.957479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2316.982471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2319.468040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2319.493453] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2320.033574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2320.050345] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2321.579771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2321.605899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2321.666600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.10@tcp -m PUT -e local_timeout -r 1 [ 2321.719208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2321.770510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2321.777486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2321.986335] LNetError: 120929:0:(lib-move.c:4801:lnet_parse()) 192.168.202.10@tcp, src 192.168.202.10@tcp: Dropping PUT (error -108 looking up sender) [ 2323.042805] LNet: Removed LNI 10.1.2.3@tcp [ 2324.077776] LNet: Removed LNI 192.168.202.10@tcp [ 2324.406893] Key type .llcrypt unregistered [ 2324.409226] Key type ._llcrypt unregistered [ 2336.290709] Key type ._llcrypt registered [ 2336.293181] Key type .llcrypt registered [ 2337.176951] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:19:22 (1787192362) [ 2337.625932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.638361] alg: No test for adler32 (adler32-zlib) [ 2338.503744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.797469] LNet: 122244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.241515] Key type .llcrypt unregistered [ 2339.246067] Key type ._llcrypt unregistered [ 2345.668070] Key type ._llcrypt registered [ 2345.669626] Key type .llcrypt registered [ 2346.028704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2346.050910] alg: No test for adler32 (adler32-zlib) [ 2346.990742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2347.036153] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2354.346641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2354.495739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2354.969031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2355.174324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2355.226651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2355.237085] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2355.270623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2355.296505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2355.321785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2355.343262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2355.368869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2355.393505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2355.408995] LNet: 122697:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2355.425538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2355.447081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2355.472642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2355.495156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2355.530779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2355.557403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2356.052861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2356.284459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2356.310210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2356.338580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2356.361371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2356.389226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2356.425574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2356.463458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2356.488215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2356.509312] LNet: 122815:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.524643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2356.547370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2356.572876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2356.602304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2356.637852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2356.666596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2357.178839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2357.358824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.388921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2357.426856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.458723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.497181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.534322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2357.568093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2357.600859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2357.614371] LNet: 122932:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2357.627520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2357.649219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2357.680947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2357.717258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2357.749032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2357.788278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2358.409380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2358.596930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2358.621104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2358.648366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2358.677747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2358.703810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2358.730907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.755884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.782353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.809868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.839983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.865896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.895632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.920587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.948621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2359.233455] LNet: 123099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.238471] LNetError: 123099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.252949] LNet: Removed LNI 192.168.202.10@tcp [ 2359.514612] Key type .llcrypt unregistered [ 2359.516250] Key type ._llcrypt unregistered [ 2377.104824] Key type ._llcrypt registered [ 2377.108233] Key type .llcrypt registered [ 2378.605297] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:20:03 (1787192403) [ 2379.071459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2379.102076] alg: No test for adler32 (adler32-zlib) [ 2380.040224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.485263] LNet: 123986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2381.130343] Key type .llcrypt unregistered [ 2381.133578] Key type ._llcrypt unregistered [ 2388.433113] Key type ._llcrypt registered [ 2388.437182] Key type .llcrypt registered [ 2388.796201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2388.813386] alg: No test for adler32 (adler32-zlib) [ 2389.877387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2389.931542] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2398.222761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.110@tcp [ 2407.870140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.926913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2407.950869] LNet: Added LNI 192.168.202.10@tcp1 [8/256/0/180] [ 2409.089410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2409.686549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.110@tcp [ 2409.974732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2410.016510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2410.052518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2410.079459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2410.102324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2410.133827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2410.169726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2410.216669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2410.247491] LNet: 124644:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2410.279884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2410.353367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2410.396632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2410.411425] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.110@tcp [ 2410.443502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2410.460898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2410.479815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2411.126561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2411.405990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2411.443977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2411.479348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2411.505253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2411.542857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2411.583212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2411.609841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2411.670581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2411.686760] LNet: 124761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2411.707626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2411.750138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2411.847320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2411.891210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2411.943643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2412.016568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2412.758433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.110@tcp [ 2412.965368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2412.991803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2413.016747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2413.047525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2413.080551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2413.112537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2413.140624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2413.173703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2413.190645] LNet: 124878:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2413.209099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2413.234422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2413.274629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2413.304562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2413.347677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2413.376571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2414.150308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2414.549210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2414.582244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2414.637934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2414.685370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2414.716620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2414.740738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2414.765352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2414.789802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2414.814511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2414.840672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2414.865697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2414.890711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2414.934572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2414.983221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2421.639423] LNet: 125081:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.650303] LNetError: 125081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.660316] LNet: Removed LNI 192.168.202.10@tcp [ 2421.675161] LNet: Removed LNI 192.168.202.10@tcp1 [ 2422.026772] Key type .llcrypt unregistered [ 2422.028838] Key type ._llcrypt unregistered [ 2442.216241] Key type ._llcrypt registered [ 2442.218039] Key type .llcrypt registered [ 2443.744349] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:21:08 (1787192468) [ 2444.436458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.457133] alg: No test for adler32 (adler32-zlib) [ 2445.483891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.544143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.582985] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2446.185232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2446.217804] LNet: Added LNI 192.168.202.10@tcp2 [8/256/0/180] [ 2446.980344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.10@tcp [ 2447.065422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2448.095633] LNet: Removed LNI 192.168.202.10@tcp [ 2453.782448] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:21:18 (1787192478) [ 2455.053534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2457.040485] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:21:21 (1787192481) [ 2465.209800] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:21:29 (1787192489) [ 2472.021442] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:21:37 (1787192497) [ 2480.804784] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:21:45 (1787192505) [ 2487.242237] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:21:51 (1787192511) [ 2494.349464] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:21:59 (1787192519) [ 2502.053868] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:22:06 (1787192526) [ 2509.044679] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:22:13 (1787192533) [ 2517.084780] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:22:21 (1787192541) [ 2523.450384] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:22:28 (1787192548) [ 2530.213518] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:22:35 (1787192555) [ 2537.789401] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:22:42 (1787192562) [ 2543.988599] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:22:49 (1787192569) [ 2552.296397] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:22:56 (1787192576) [ 2560.676249] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:23:04 (1787192584) [ 2568.187985] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:23:12 (1787192592) [ 2575.663714] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:23:20 (1787192600) [ 2583.341636] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:23:28 (1787192608) [ 2583.804338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2583.843266] LNet: 136775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.850551] LNetError: 136775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2583.874219] LNet: Removed LNI 192.168.202.10@tcp2 [ 2583.969886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2584.223953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2584.256516] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2585.693296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2585.724375] LNet: 136918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.731577] LNetError: 136918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.751143] LNet: Removed LNI 192.168.202.10@tcp [ 2585.769381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2585.820767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2585.851164] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2592.652553] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:23:37 (1787192617) [ 2599.174700] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:23:43 (1787192623) [ 2606.787515] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:23:51 (1787192631) [ 2613.206143] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:23:58 (1787192638) [ 2620.537841] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:24:05 (1787192645) [ 2630.170706] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:24:14 (1787192654) [ 2640.302921] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:24:24 (1787192664) [ 2648.272716] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:24:32 (1787192672) [ 2656.335250] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:24:41 (1787192681) [ 2665.082965] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:24:49 (1787192689) [ 2672.931935] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:24:57 (1787192697) [ 2680.821248] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:25:05 (1787192705) [ 2688.745601] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2385 sec ========== 22:25:13 (1787192713) [ 2689.372361] LNet: 144385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2689.389154] LNetError: 144385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2689.402593] LNet: Removed LNI 192.168.202.10@tcp [ 2690.216644] Key type .llcrypt unregistered [ 2690.219113] Key type ._llcrypt unregistered [ 2707.567599] Key type ._llcrypt registered [ 2707.578076] Key type .llcrypt registered [ 2708.117709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2708.132443] alg: No test for adler32 (adler32-zlib) [ 2709.522886] Lustre: Lustre: Build Version: 2.17.57_44_gafaaa93 [ 2709.803403] LNet: Added LNI 192.168.202.10@tcp [8/256/0/180] [ 2711.567363] Key type lgssc registered [ 2713.445843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2784.535498] Lustre: Mounted lustre-client - version 2.17.57_44_gafaaa93 [ 2790.404452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2801.838092] Lustre: Unmounted lustre-client [ 2833.130310] Key type lgssc unregistered [ 2833.363747] LNet: 147347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2833.384298] LNetError: 147347:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2833.414830] LNet: Removed LNI 192.168.202.10@tcp [ 2834.411359] Key type .llcrypt unregistered [ 2834.415533] Key type ._llcrypt unregistered