[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466587796 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002343] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011000] SELinux: Initializing. [ 0.011540] *** VALIDATE selinux *** [ 0.023598] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029240] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034104] *** VALIDATE tmpfs *** [ 0.036487] *** VALIDATE proc *** [ 0.038067] *** VALIDATE cgroup *** [ 0.039013] *** VALIDATE cgroup2 *** [ 0.041219] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.042149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.044007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.045031] Spectre V2 : User space: Vulnerable [ 0.046014] Speculative Store Bypass: Vulnerable [ 0.050050] debug: unmapping init [mem 0xffffffff8e059000-0xffffffff8e060fff] [ 0.052334] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.053763] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.054023] ... version: 2 [ 0.055014] ... bit width: 48 [ 0.056016] ... generic registers: 4 [ 0.057014] ... value mask: 0000ffffffffffff [ 0.058018] ... max period: 00007fffffffffff [ 0.059016] ... fixed-purpose events: 3 [ 0.060016] ... event mask: 000000070000000f [ 0.061303] rcu: Hierarchical SRCU implementation. [ 0.063689] smp: Bringing up secondary CPUs ... [ 0.065133] x86: Booting SMP configuration: [ 0.066015] .... node #0, CPUs: #1 #2 #3 [ 0.078066] smp: Brought up 1 node, 4 CPUs [ 0.080013] smpboot: Max logical packages: 1 [ 0.081013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.154000] node 0 deferred pages initialised in 71ms [ 0.159319] devtmpfs: initialized [ 0.160247] x86/mm: Memory block size: 128MB [ 0.162837] gcov: version magic: 0x41383552 [ 0.165147] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166139] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.167308] pinctrl core: initialized pinctrl subsystem [ 0.168381] [ 0.169010] ************************************************************* [ 0.170019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171015] ** ** [ 0.172015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.173017] ** ** [ 0.174016] ** This means that this kernel is built to expose internal ** [ 0.175015] ** IOMMU data structures, which may compromise security on ** [ 0.176013] ** your system. ** [ 0.177015] ** ** [ 0.178017] ** If you see this message and you are not debugging the ** [ 0.179016] ** kernel, report this immediately to your vendor! ** [ 0.180079] ** ** [ 0.181013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182014] ************************************************************* [ 0.183958] NET: Registered protocol family 16 [ 0.184764] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.185063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.186065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.187983] cpuidle: using governor menu [ 0.188000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.195776] PCI: Using configuration type 1 for base access [ 0.199187] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.209111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.210015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.213035] cryptd: max_cpu_qlen set to 1000 [ 0.216260] ACPI: Added _OSI(Module Device) [ 0.217022] ACPI: Added _OSI(Processor Device) [ 0.218015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.219015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223427] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226472] ACPI: Interpreter enabled [ 0.227000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227013] ACPI: Using IOAPIC for interrupt routing [ 0.228000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.232375] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.244000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.245037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.248019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.252088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.259676] acpiphp: Slot [2] registered [ 0.261125] acpiphp: Slot [5] registered [ 0.263113] acpiphp: Slot [6] registered [ 0.265158] acpiphp: Slot [3] registered [ 0.267165] acpiphp: Slot [4] registered [ 0.269149] acpiphp: Slot [7] registered [ 0.271115] acpiphp: Slot [8] registered [ 0.273114] acpiphp: Slot [9] registered [ 0.275224] acpiphp: Slot [10] registered [ 0.277184] acpiphp: Slot [11] registered [ 0.280109] acpiphp: Slot [12] registered [ 0.281103] acpiphp: Slot [13] registered [ 0.283146] acpiphp: Slot [14] registered [ 0.285109] acpiphp: Slot [15] registered [ 0.287277] acpiphp: Slot [16] registered [ 0.290118] acpiphp: Slot [17] registered [ 0.292127] acpiphp: Slot [18] registered [ 0.293000] acpiphp: Slot [19] registered [ 0.294206] acpiphp: Slot [20] registered [ 0.296124] acpiphp: Slot [21] registered [ 0.298126] acpiphp: Slot [22] registered [ 0.301128] acpiphp: Slot [23] registered [ 0.302079] acpiphp: Slot [24] registered [ 0.304120] acpiphp: Slot [25] registered [ 0.311050] acpiphp: Slot [26] registered [ 0.314298] acpiphp: Slot [27] registered [ 0.316122] acpiphp: Slot [28] registered [ 0.318242] acpiphp: Slot [29] registered [ 0.320113] acpiphp: Slot [30] registered [ 0.322118] acpiphp: Slot [31] registered [ 0.324094] PCI host bridge to bus 0000:00 [ 0.325024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.330025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.338024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.344053] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.348031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.351338] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.354441] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.365179] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.382523] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.396573] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.399553] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.401016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.404013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.406488] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.422213] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.427049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.430759] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.437128] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.445017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.447023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.453000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.462014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.471017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.493015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503689] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.525014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.548078] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.559491] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.562268] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.565557] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.568398] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.571337] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.575159] iommu: Default domain type: Passthrough [ 0.576531] SCSI subsystem initialized [ 0.577164] ACPI: bus type USB registered [ 0.578199] usbcore: registered new interface driver usbfs [ 0.581317] usbcore: registered new interface driver hub [ 0.585108] usbcore: registered new device driver usb [ 0.587175] pps_core: LinuxPPS API ver. 1 registered [ 0.590011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.593126] PTP clock support registered [ 0.597227] EDAC MC: Ver: 3.0.0 [ 0.599140] PCI: Using ACPI for IRQ routing [ 0.601677] NetLabel: Initializing [ 0.603011] NetLabel: domain hash size = 128 [ 0.604009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.607089] NetLabel: unlabeled traffic allowed by default [ 0.608285] vgaarb: loaded [ 0.610323] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.612017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.618378] clocksource: Switched to clocksource kvm-clock [ 0.731449] VFS: Disk quotas dquot_6.6.0 [ 0.733242] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.736131] *** VALIDATE ramfs *** [ 0.737432] *** VALIDATE hugetlbfs *** [ 0.739164] pnp: PnP ACPI init [ 0.741497] pnp: PnP ACPI: found 6 devices [ 0.758602] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.762409] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.764801] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.767291] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.769986] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772542] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.775885] NET: Registered protocol family 2 [ 0.778620] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.783542] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.788054] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.794541] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.799056] TCP: Hash tables configured (established 65536 bind 65536) [ 0.802297] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.805377] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.808477] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.813336] NET: Registered protocol family 1 [ 0.816111] RPC: Registered named UNIX socket transport module. [ 0.818536] RPC: Registered udp transport module. [ 0.820352] RPC: Registered tcp transport module. [ 0.822398] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.825446] NET: Registered protocol family 44 [ 0.827042] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.829755] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.831733] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.834258] PCI: CLS 0 bytes, default 64 [ 0.836123] Unpacking initramfs... [ 4.133394] debug: unmapping init [mem 0xffff9c433cc64000-0xffff9c433ffcffff] [ 4.145260] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.150718] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.156066] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.240249] Initialise system trusted keyrings [ 6.246894] Key type blacklist registered [ 6.255325] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.278926] zbud: loaded [ 6.285276] *** VALIDATE nfs *** [ 6.287321] *** VALIDATE nfs4 *** [ 6.299989] pstore: using deflate compression [ 6.313653] Platform Keyring initialized [ 6.699381] NET: Registered protocol family 38 [ 6.708155] Key type asymmetric registered [ 6.714302] Asymmetric key parser 'x509' registered [ 6.717678] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.729454] io scheduler mq-deadline registered [ 6.734529] io scheduler kyber registered [ 6.737669] io scheduler bfq registered [ 6.741252] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.750248] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.759438] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.765439] ACPI: Power Button [PWRF] [ 6.782860] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.805312] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.842721] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.905165] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.968249] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.997384] Non-volatile memory driver v1.3 [ 7.001701] Linux agpgart interface v0.103 [ 7.175199] virtio_blk virtio1: [vda] 136528 512-byte logical blocks (69.9 MB/66.7 MiB) [ 7.192154] vda: detected capacity change from 0 to 69902336 [ 7.246816] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.258299] vdb: detected capacity change from 0 to 1073741824 [ 7.307491] libphy: Fixed MDIO Bus: probed [ 7.377422] usbcore: registered new interface driver usbserial_generic [ 7.387124] usbserial: USB Serial support registered for generic [ 7.400422] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.423801] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.428112] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.437796] mousedev: PS/2 mouse device common for all mice [ 7.444059] rtc_cmos 00:05: RTC can wake from S4 [ 7.450192] rtc_cmos 00:05: registered as rtc0 [ 7.453337] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.455312] intel_pstate: CPU model not supported [ 7.458821] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.466393] hid: raw HID events driver (C) Jiri Kosina [ 7.475930] usbcore: registered new interface driver usbhid [ 7.477953] usbhid: USB HID core driver [ 7.481394] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.488790] drop_monitor: Initializing network drop monitor service [ 7.493344] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.494413] Initializing XFRM netlink socket [ 7.496101] NET: Registered protocol family 10 [ 7.511435] Segment Routing with IPv6 [ 7.515233] NET: Registered protocol family 17 [ 7.518853] mpls_gso: MPLS GSO support [ 7.531240] RAS: Correctable Errors collector initialized. [ 7.533412] AVX version of gcm_enc/dec engaged. [ 7.541508] AES CTR mode by8 optimization enabled [ 7.761628] sched_clock: Marking stable (7761495614, 0)->(9014770439, -1253274825) [ 7.769820] registered taskstats version 1 [ 7.775708] Loading compiled-in X.509 certificates [ 7.781429] zswap: loaded using pool lzo/zbud [ 7.871281] Key type big_key registered [ 7.903537] Key type encrypted registered [ 7.909280] ima: No TPM chip found, activating TPM-bypass! [ 7.913472] ima: Allocated hash algorithm: sha1 [ 7.920333] ima: No architecture policies found [ 7.923317] evm: Initialising EVM extended attributes: [ 7.926775] evm: security.selinux [ 7.931517] evm: security.ima [ 7.941067] hrtimer: interrupt took 6180167 ns [ 7.952780] evm: security.capability [ 7.958469] evm: HMAC attrs: 0x1 [ 7.968788] rtc_cmos 00:05: setting system clock to 2026-06-12 01:42:12 UTC (1781228532) [ 7.989487] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 8.004519] debug: unmapping init [mem 0xffffffff8dd82000-0xffffffff8e058fff] [ 8.022156] Write protecting the kernel read-only data: 28672k [ 8.043510] debug: unmapping init [mem 0xffffffff8c403000-0xffffffff8c5fffff] [ 8.051736] debug: unmapping init [mem 0xffffffff8cd14000-0xffffffff8cdfffff] [ 8.223985] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.244496] systemd[1]: Detected virtualization kvm. [ 8.247066] systemd[1]: Detected architecture x86-64. [ 8.253488] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.347995] systemd[1]: No hostname configured. [ 8.352295] systemd[1]: Set hostname to . [ 8.358305] random: systemd: uninitialized urandom read (16 bytes read) [ 8.369550] systemd[1]: Initializing machine ID from random generator. [ 8.533047] random: ln: uninitialized urandom read (6 bytes read) [ 8.850284] random: systemd: uninitialized urandom read (16 bytes read) [ 8.855899] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.881662] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 8.898439] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.213890] device-mapper: uevent: version 1.0.3 [ 11.223863] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.983733] virtio_net virtio0 ens2: renamed from eth0 [ 14.427563] scsi host0: ata_piix [ 14.630354] scsi host1: ata_piix [ 14.634506] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 14.647293] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.511619] random: fast init done [ 19.481538] random: crng init done [ 19.486269] random: 7 urandom warning(s) missed due to ratelimiting [ 20.579392] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 22.801275] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.899488] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.072577] SELinux: Disabled at runtime. [ 29.283140] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.317286] systemd[1]: Detected virtualization kvm. [ 29.324175] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.917303] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.928422] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.935754] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.942289] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.946783] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.957259] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.976142] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Started Crea[ 31.760341] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS te list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 32.541220] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 33.578269] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.671987] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.194634] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.326445] EDAC sbridge: Ver: 1.1.2 [ 36.299300] Key type dns_resolver registered [ 37.005817] NFS: Registering the id_resolver key type [ 37.008074] Key type id_resolver registered [ 37.009043] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-client login: [ 96.934608] libcfs: loading out-of-tree module taints kernel. [ 97.002319] Key type ._llcrypt registered [ 97.008631] Key type .llcrypt registered [ 97.552949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 97.571993] alg: No test for adler32 (adler32-zlib) [ 99.115467] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 99.958937] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 101.729245] Key type lgssc registered [ 103.530941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 234.019852] Lustre: Mounted lustre-client [ 238.671282] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.498448] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.470340] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing yml_node [ 259.557044] Lustre: lustre-OST0000-osc-ffff9c4390ec4000: disconnect after 24s idle [ 263.381412] Lustre: DEBUG MARKER: Client: 2.17.53.64 [ 266.302458] Lustre: DEBUG MARKER: MDS: 2.17.53.64 [ 268.737557] Lustre: DEBUG MARKER: OSS: 2.17.53.64 [ 270.336508] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jun 11 21:46:33 EDT 2026 [ 285.492572] Lustre: Unmounted lustre-client [ 314.114491] Key type lgssc unregistered [ 314.485416] LNet: 8399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.489884] LNetError: 8399:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 314.520922] LNet: Removed LNI 192.168.203.16@tcp [ 315.278171] Key type .llcrypt unregistered [ 315.280458] Key type ._llcrypt unregistered [ 340.243805] Key type ._llcrypt registered [ 340.247961] Key type .llcrypt registered [ 340.474906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 340.499845] alg: No test for adler32 (adler32-zlib) [ 341.441527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 341.500740] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 341.521263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 345.090454] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 345.375884] LNet: 9283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.397344] LNetError: 9283:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 345.410470] LNet: Removed LNI 192.168.203.16@tcp [ 346.038528] Key type .llcrypt unregistered [ 346.040792] Key type ._llcrypt unregistered [ 352.431907] Key type ._llcrypt registered [ 352.437071] Key type .llcrypt registered [ 353.544913] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:47:56 (1781228876) [ 353.943838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 353.968890] alg: No test for adler32 (adler32-zlib) [ 354.837339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.912167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 354.934360] LNet: 9696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 354.957746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.060234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 355.079786] LNet: 9703:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 362.582606] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:48:05 (1781228885) [ 362.688549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.732209] LNet: 10274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.755448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.833658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 369.466551] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:48:12 (1781228892) [ 369.637270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.662477] LNet: 10864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.719859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.783812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 376.002088] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:48:19 (1781228899) [ 376.158408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.193300] LNet: 11450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.228102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.314907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 382.044907] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:48:25 (1781228905) [ 382.140356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.176335] LNet: 12028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.196513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.314695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 382.621885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 388.645874] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:48:31 (1781228911) [ 388.839189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.908483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.986709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 389.510786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.066333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 396.423716] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:48:39 (1781228919) [ 396.685632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.705705] LNet: 13222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.711621] LNet: 13222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 396.736882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.810832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 402.957639] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:48:46 (1781228926) [ 403.132552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.213676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.296125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 403.352588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.019874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 404.076883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 404.685945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 404.725392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.361342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 405.400207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 405.803855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 405.840552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.342356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 406.409274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.944821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 406.986131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 407.541190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 407.567821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 408.055759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 408.102245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 408.651342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 408.725135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 409.313491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 409.370136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.040642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 410.107145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 410.544527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 410.593776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.181345] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:49:00 (1781228940) [ 417.347162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.410910] LNet: 14486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.422809] LNet: 14486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 417.459692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.635460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 417.708779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 424.753567] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:49:08 (1781228948) [ 425.069845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.112239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.215546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 425.251190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 431.829581] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:49:14 (1781228954) [ 432.011981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.091845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.184591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 432.220153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 439.383115] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:49:22 (1781228962) [ 439.520430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.551352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.625767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 439.659353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 445.863787] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:49:29 (1781228969) [ 446.039162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.106783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.174486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 446.207539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 452.339629] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:49:35 (1781228975) [ 452.446252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.481230] LNet: 17415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 452.486296] LNet: 17415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 452.517889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.599680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 452.628848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 458.645274] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:49:41 (1781228981) [ 458.750891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.800821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.929452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.406911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.979968] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:49:49 (1781228989) [ 466.255825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.308054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.574657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.803242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 473.130666] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:49:56 (1781228996) [ 473.323884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.402280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.529206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.745889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.901218] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:50:04 (1781229004) [ 481.064618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.133279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.731964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 482.075740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.528889] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:50:11 (1781229011) [ 488.720914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.802239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.894108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.965470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.954450] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:50:18 (1781229018) [ 495.088336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.154363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.242381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.298760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.348737] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:50:26 (1781229026) [ 503.440246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.468197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.534477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 510.817052] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:50:33 (1781229033) [ 511.100218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.208875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.327479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 517.712852] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:50:40 (1781229040) [ 517.838313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.878982] LNet: 22852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.893667] LNet: 22852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 517.933600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.054968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 518.611794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 526.360994] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:50:49 (1781229049) [ 526.492192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.601376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.710904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 526.742448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 533.626464] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:50:56 (1781229056) [ 533.746382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.796950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.875477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 533.927659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 541.331265] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:51:04 (1781229064) [ 541.511741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.549626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.591519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 541.613695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 547.068577] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:51:10 (1781229070) [ 547.144909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.203567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.306432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 547.336893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 553.592882] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:51:16 (1781229076) [ 553.828173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.885464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.950175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 554.015957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 554.042745] LNetError: 25807:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 554.083337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 554.106583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 560.173385] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:51:23 (1781229083) [ 560.293801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.334636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.416516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.V0aedPWfzL [ 560.479684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.V0aedPWfzL [ 560.532271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.V0aedPWfzL [ 566.359440] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:51:29 (1781229089) [ 566.442348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.477782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.583931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 566.631985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 566.676509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 572.717429] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:51:35 (1781229095) [ 572.824316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.885848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.991879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 576.017081] LNet: Message forwarding will be enabled in 2s [ 579.104197] LNet: Message forwarding enabled [ 579.499891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 579.523775] LNet: Message forwarding will stop in 2s to 6s [ 583.974470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 584.631731] LNet: Message forwarding disabled [ 586.316373] LNet: Message forwarding will be enabled in 2s [ 589.794015] LNet: Message forwarding enabled [ 593.559779] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:51:56 (1781229116) [ 593.715574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.348633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.500150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 602.654179] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:52:05 (1781229125) [ 603.034540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.105676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.187629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 603.221991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 603.255962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 603.290755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 603.339958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 603.379536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 603.422251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 603.464249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 603.498239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 603.546784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 603.568388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 603.589724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 603.627900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 603.680048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 603.714882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 603.746579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 603.773672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 603.818824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 603.868196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 603.883234] LNetError: 28878:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 609.594682] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:52:12 (1781229132) [ 609.734657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.787768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.885880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.699771] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:52:19 (1781229139) [ 618.418405] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg316-server [ 620.408734] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:52:23 (1781229143) [ 620.504800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.569694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.667813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 620.723037] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 622.840349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[17-21]@tcp [ 622.895861] LNet: 30595:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 622.961023] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 622.984764] LNetError: 30383:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.18@tcp has gone from up to down [ 623.989681] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 624.019979] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 625.020261] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 625.033919] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 627.049083] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.20@tcp is being used as a gateway but routing feature is not turned on [ 627.064500] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 630.043591] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:52:33 (1781229153) [ 631.157146] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.19@tcp is being used as a gateway but routing feature is not turned on [ 631.181844] LNetError: 30383:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 631.917981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg316-server [ 633.719736] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:52:36 (1781229156) [ 633.880955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.913435] LNetError: 31527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.979734] LNet: Removed LNI 192.168.203.16@tcp [ 634.011688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.072920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 634.093052] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 634.935359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 634.960792] LNet: 31647:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 634.971357] LNet: 31647:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 635.005705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 635.008676] LNetError: 31533:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.17@tcp has gone from up to down [ 635.034508] LNetError: 31533:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 641.248197] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:52:44 (1781229164) [ 641.397972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.444644] LNetError: 32234:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.487816] LNet: Removed LNI 192.168.203.16@tcp [ 641.566058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.657887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 641.700687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.727887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.769719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 641.812748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.849528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.892579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 641.962973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.009177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.098230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.194290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.223670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.268793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 642.353183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.376732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.460450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.558105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.624162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.684888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 642.781416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.821851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.882257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.943910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.974553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.043243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 643.143842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.177354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.246582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.350197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.387568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.484397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 643.550647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.600859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.684272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 649.751432] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:52:52 (1781229172) [ 651.353445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg316-server [ 652.883482] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:52:56 (1781229176) [ 654.937209] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg316-server [ 656.529826] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:52:59 (1781229179) [ 656.676889] LNet: 33631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.682623] LNet: 33631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 25 previous similar messages [ 657.228899] Key type .llcrypt unregistered [ 657.230560] Key type ._llcrypt unregistered [ 663.683071] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 663.828132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 663.836344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 663.841817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 666.529842] Key type ._llcrypt registered [ 666.532907] Key type .llcrypt registered [ 666.738749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 666.753130] alg: No test for adler32 (adler32-zlib) [ 667.693501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.772931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.792201] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 668.550413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 668.568605] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 670.940354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 670.972450] LNet: Removed LNI 10.1.2.3@tcp [ 671.126516] LNet: 34354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.143355] LNetError: 34354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.165538] LNet: Removed LNI 192.168.203.16@tcp [ 671.856730] Key type .llcrypt unregistered [ 671.859453] Key type ._llcrypt unregistered [ 681.733498] Key type ._llcrypt registered [ 681.735753] Key type .llcrypt registered [ 683.026606] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:53:26 (1781229206) [ 683.634801] Key type .llcrypt unregistered [ 683.638306] Key type ._llcrypt unregistered [ 690.998880] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 691.247550] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 691.295863] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 691.309699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 694.091445] Key type ._llcrypt registered [ 694.093047] Key type .llcrypt registered [ 694.444156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 694.479193] alg: No test for adler32 (adler32-zlib) [ 695.656921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.828406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.904149] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 697.334063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 697.396393] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 701.771818] LNet: 35842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.786341] LNetError: 35842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.799585] LNet: Removed LNI 192.168.203.16@tcp [ 701.825172] LNet: Removed LNI 10.1.2.3@tcp [ 702.828451] Key type .llcrypt unregistered [ 702.834172] Key type ._llcrypt unregistered [ 716.041912] Key type ._llcrypt registered [ 716.045033] Key type .llcrypt registered [ 717.132548] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:54:00 (1781229240) [ 717.864834] Key type .llcrypt unregistered [ 717.868331] Key type ._llcrypt unregistered [ 724.036824] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.203171] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.211078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.215250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.920157] Key type ._llcrypt registered [ 726.922048] Key type .llcrypt registered [ 727.176841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.196789] alg: No test for adler32 (adler32-zlib) [ 728.111045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.168898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 728.219171] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 730.259665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 730.277902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 732.452306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 732.484321] LNet: Removed LNI 10.1.2.3@tcp [ 732.507356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 732.542926] LNet: Removed LNI 10.1.2.31@tcp [ 732.557480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 732.592339] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 733.278854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 733.300864] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 734.562666] LNet: 37761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.578870] LNetError: 37761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 734.590865] LNet: Removed LNI 10.1.2.3@tcp [ 734.616510] LNet: Removed LNI 10.1.2.31@tcp [ 735.511475] Key type .llcrypt unregistered [ 735.514203] Key type ._llcrypt unregistered [ 749.067496] Key type ._llcrypt registered [ 749.070725] Key type .llcrypt registered [ 750.412699] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:54:33 (1781229273) [ 750.822740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 750.845852] alg: No test for adler32 (adler32-zlib) [ 751.833322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.915612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.944670] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 753.021703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 753.046722] LNet: 38715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.054435] LNetError: 38715:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.123438] LNet: Removed LNI 192.168.203.16@tcp [ 753.183480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 753.287870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.16@tcp [ 753.337033] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 754.255466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.16@tcp [ 754.297844] LNet: Removed LNI 192.168.203.16@tcp [ 754.428312] LNet: 38850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.149499] Key type .llcrypt unregistered [ 755.153178] Key type ._llcrypt unregistered [ 767.221426] Key type ._llcrypt registered [ 767.224906] Key type .llcrypt registered [ 768.719304] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:54:51 (1781229291) [ 770.457980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg316-server [ 772.177238] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:54:55 (1781229295) [ 772.673435] Key type .llcrypt unregistered [ 772.680260] Key type ._llcrypt unregistered [ 778.902743] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.077558] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.086672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.090149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.704111] Key type ._llcrypt registered [ 781.707446] Key type .llcrypt registered [ 781.929269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.939468] alg: No test for adler32 (adler32-zlib) [ 782.927976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.022116] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 783.701162] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 785.845214] LNet: 40681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.850433] LNetError: 40681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.856581] LNet: Removed LNI 192.168.203.16@tcp1 [ 785.871925] LNet: Removed LNI 10.1.2.3@tcp2 [ 786.378531] Key type .llcrypt unregistered [ 786.380368] Key type ._llcrypt unregistered [ 799.407136] Key type ._llcrypt registered [ 799.411923] Key type .llcrypt registered [ 800.881493] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:55:23 (1781229323) [ 801.431938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.454142] alg: No test for adler32 (adler32-zlib) [ 802.387412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.442030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.485791] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 803.366267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.392832] LNet: 41650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.406434] LNetError: 41650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.429535] LNet: Removed LNI 192.168.203.16@tcp [ 803.464409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.559272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 803.594884] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 810.678534] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:55:33 (1781229333) [ 810.861204] LNet: 42240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.873105] LNetError: 42240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.916092] LNet: Removed LNI 192.168.203.16@tcp [ 811.429542] Key type .llcrypt unregistered [ 811.432327] Key type ._llcrypt unregistered [ 817.910257] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 818.130319] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 818.143782] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 818.153673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.762906] Key type ._llcrypt registered [ 820.765746] Key type .llcrypt registered [ 820.937216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.947253] alg: No test for adler32 (adler32-zlib) [ 821.883865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.951307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 821.984523] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 822.829974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.848470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.825501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.843832] LNet: 42981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.853866] LNetError: 42981:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.860317] LNet: Removed LNI 192.168.203.16@tcp [ 824.869790] LNet: Removed LNI 10.1.2.3@tcp [ 824.893176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.951359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.981623] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 824.991085] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.411789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.435649] LNet: 43008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.442593] LNetError: 43008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.447512] LNetError: 43008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 825.453827] LNet: Removed LNI 192.168.203.16@tcp [ 825.466774] LNet: Removed LNI 10.1.2.3@tcp [ 825.487309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.597821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 825.616923] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 825.622779] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.059417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 826.080899] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 826.893287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.908658] LNet: 43170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.914048] LNetError: 43170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.918692] LNetError: 43170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 826.923960] LNet: Removed LNI 192.168.203.16@tcp [ 826.926740] LNet: Removed LNI 10.1.2.3@tcp [ 826.935922] LNet: Removed LNI 192.168.203.16@tcp2 [ 826.961393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 827.043538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 827.076497] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 827.079810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 827.084239] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 834.469700] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:55:57 (1781229357) [ 834.577387] LNet: 43761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.586536] LNetError: 43761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.592107] LNetError: 43761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 834.598073] LNet: Removed LNI 192.168.203.16@tcp [ 834.599856] LNet: Removed LNI 10.1.2.3@tcp [ 834.613557] LNet: Removed LNI 192.168.203.16@tcp2 [ 835.301499] Key type .llcrypt unregistered [ 835.307646] Key type ._llcrypt unregistered [ 841.908854] Key type ._llcrypt registered [ 841.911317] Key type .llcrypt registered [ 842.115956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.141084] alg: No test for adler32 (adler32-zlib) [ 843.098040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.189133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.234051] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 844.056699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.068333] LNet: 44163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.076785] LNetError: 44163:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.090975] LNet: Removed LNI 192.168.203.16@tcp [ 844.104087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.160313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.193953] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 844.677448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.700143] LNet: 44193:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.718301] LNetError: 44193:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.749740] LNet: Removed LNI 192.168.203.16@tcp [ 844.769823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.840113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.886045] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 845.403027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.430483] LNet: Removed LNI 192.168.203.16@tcp [ 845.455695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.537617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 845.573804] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 846.106257] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 846.299080] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 846.307756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 846.317979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 847.612540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.634351] LNet: 44296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.639658] LNet: 44296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 847.647841] LNetError: 44296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.654974] LNetError: 44296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 847.685124] LNet: Removed LNI 192.168.203.16@tcp [ 847.725938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.817159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 847.857948] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 854.330481] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:56:17 (1781229377) [ 854.433134] LNet: 44884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.437621] LNetError: 44884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.456173] LNet: Removed LNI 192.168.203.16@tcp [ 855.112713] Key type .llcrypt unregistered [ 855.116835] Key type ._llcrypt unregistered [ 861.598297] Key type ._llcrypt registered [ 861.606982] Key type .llcrypt registered [ 861.806269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 861.817698] alg: No test for adler32 (adler32-zlib) [ 862.779145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.862622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.908588] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 863.884352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.913487] LNet: 45291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.925319] LNetError: 45291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.947173] LNet: Removed LNI 192.168.203.16@tcp [ 863.983809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.096701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 864.127125] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 870.393819] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:56:33 (1781229393) [ 870.626681] LNet: 45883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.634139] LNetError: 45883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.649182] LNet: Removed LNI 192.168.203.16@tcp [ 871.323033] Key type .llcrypt unregistered [ 871.327112] Key type ._llcrypt unregistered [ 876.890684] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 877.060712] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 877.072763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 877.086626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 879.652892] Key type ._llcrypt registered [ 879.657881] Key type .llcrypt registered [ 879.861378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.873589] alg: No test for adler32 (adler32-zlib) [ 880.777625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.838697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.863970] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 881.754154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.793975] LNet: 46333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.804794] LNetError: 46333:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.826679] LNet: Removed LNI 192.168.203.16@tcp [ 881.849692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.923182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 881.942095] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 888.565315] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:56:51 (1781229411) [ 888.682471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.712756] LNet: 46922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.735595] LNetError: 46922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.761521] LNet: Removed LNI 192.168.203.16@tcp [ 888.799626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.944952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 894.636480] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:56:57 (1781229417) [ 894.856432] LNet: 47514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.513201] Key type .llcrypt unregistered [ 895.515288] Key type ._llcrypt unregistered [ 902.479849] Key type ._llcrypt registered [ 902.484461] Key type .llcrypt registered [ 902.767615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.806559] alg: No test for adler32 (adler32-zlib) [ 903.822781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 903.897391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 903.928962] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 904.993515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.010594] LNet: 47911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.016481] LNetError: 47911:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.031699] LNet: Removed LNI 192.168.203.16@tcp [ 905.047717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.157309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 905.216937] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 911.920962] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:57:14 (1781229434) [ 912.029345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.049675] LNet: 48500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.054586] LNetError: 48500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.077317] LNet: Removed LNI 192.168.203.16@tcp [ 912.097502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.176197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 918.101662] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:57:21 (1781229441) [ 919.700153] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 921.350415] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:57:24 (1781229444) [ 923.016859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 924.641211] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:57:27 (1781229447) [ 926.042812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 927.567518] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:57:30 (1781229450) [ 929.113207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 931.026735] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:57:34 (1781229454) [ 932.476353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 934.147588] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:57:37 (1781229457) [ 934.246169] LNet: 50548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.900758] Key type .llcrypt unregistered [ 934.903582] Key type ._llcrypt unregistered [ 941.839933] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.260780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 942.265397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 942.281924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 943.872258] Key type ._llcrypt registered [ 943.875959] Key type .llcrypt registered [ 944.207164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.231336] alg: No test for adler32 (adler32-zlib) [ 945.311583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.608592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 945.656858] LNet: Added LNI 192.168.203.16@tcp [8/512/0/180] [ 945.679653] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 945.687759] LNet: 50870:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 945.872049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.939316] LNet: 50889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.951486] LNetError: 50889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.957968] LNet: Removed LNI 192.168.203.16@tcp [ 945.959868] LNetError: 50889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.000379] LNet: Removed LNI 10.1.2.3@tcp [ 946.036217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.142550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 946.163089] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 946.241877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.305677] LNet: 50914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.317382] LNetError: 50914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.344306] LNet: Removed LNI 192.168.203.16@tcp [ 946.366841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.515692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 946.600355] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 946.618572] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.624910] LNet: 50922:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 946.740986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.766155] LNet: 50940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.775585] LNetError: 50940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.782621] LNet: Removed LNI 192.168.203.16@tcp [ 946.792516] LNetError: 50940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.817808] LNet: Removed LNI 10.1.2.3@tcp [ 946.839513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.892290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 946.920587] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 946.929206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.935292] LNet: 50948:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 947.141654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.187886] LNet: 50974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.191872] LNetError: 50974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.206905] LNet: Removed LNI 192.168.203.16@tcp [ 947.210260] LNetError: 50974:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.226463] LNet: Removed LNI 10.1.2.3@tcp [ 947.293163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.383720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.422488] LNet: Added LNI 192.168.203.16@tcp [8/256/0/182] [ 947.432934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 947.627795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.667146] LNet: 51002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.687612] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.697792] LNet: Removed LNI 192.168.203.16@tcp [ 947.701669] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.738790] LNet: Removed LNI 10.1.2.3@tcp [ 947.798922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.944409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.991681] LNet: Added LNI 192.168.203.16@tcp [8/256/0/182] [ 948.009225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 948.023332] LNet: 51010:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.212107] LNet: Removed LNI 192.168.203.16@tcp [ 948.223921] LNet: Removed LNI 10.1.2.3@tcp [ 949.068871] Key type .llcrypt unregistered [ 949.080011] Key type ._llcrypt unregistered [ 961.655594] Key type ._llcrypt registered [ 961.659467] Key type .llcrypt registered [ 962.727951] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:58:06 (1781229486) [ 962.998672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.015967] alg: No test for adler32 (adler32-zlib) [ 963.905103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 963.940069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 963.954584] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 964.581319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.16 [ 964.611458] LNet: 51986:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 964.620193] LNet: Removed LNI 192.168.203.16@tcp [ 971.076387] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:58:14 (1781229494) [ 971.387251] LNet: 52556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.396621] LNetError: 52556:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.416306] LNet: Removed LNI 192.168.203.16@tcp [ 972.136339] Key type .llcrypt unregistered [ 972.138882] Key type ._llcrypt unregistered [ 979.026064] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.185708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 981.910333] Key type ._llcrypt registered [ 981.913250] Key type .llcrypt registered [ 982.091445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 982.111834] alg: No test for adler32 (adler32-zlib) [ 983.394545] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 989.010996] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:58:32 (1781229512) [ 989.135769] LNet: 53463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.142709] LNetError: 53463:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 989.181875] LNet: Removed LNI 10.1.2.3@tcp [ 989.774666] Key type .llcrypt unregistered [ 989.777760] Key type ._llcrypt unregistered [ 995.487219] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 995.678912] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 995.691094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 995.699904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 998.512606] Key type ._llcrypt registered [ 998.514768] Key type .llcrypt registered [ 998.750503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.763060] alg: No test for adler32 (adler32-zlib) [ 999.747747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1005.733818] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:58:49 (1781229529) [ 1007.011878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1008.634209] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:58:51 (1781229531) [ 1008.860199] LNet: 54652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.866443] LNetError: 54652:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1008.884704] LNet: Removed LNI 10.1.2.3@tcp [ 1009.455656] Key type .llcrypt unregistered [ 1009.458807] Key type ._llcrypt unregistered [ 1015.272285] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1015.467762] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1015.480590] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1015.487299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1018.169655] Key type ._llcrypt registered [ 1018.173891] Key type .llcrypt registered [ 1018.346688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1018.367624] alg: No test for adler32 (adler32-zlib) [ 1019.347076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1019.431689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1028.077749] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:59:11 (1781229551) [ 1028.583563] LNet: 55881:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.597225] LNetError: 55881:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1028.619338] LNet: Removed LNI 10.1.2.3@tcp [ 1029.179563] Key type .llcrypt unregistered [ 1029.184233] Key type ._llcrypt unregistered [ 1035.731190] Key type ._llcrypt registered [ 1035.734104] Key type .llcrypt registered [ 1035.991916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.040651] alg: No test for adler32 (adler32-zlib) [ 1037.085650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1037.146364] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1044.827662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1045.430510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1045.468094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1045.528441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1045.560490] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1045.869587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1045.911386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1045.951889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.008251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.049730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.083696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.136414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.164467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.188640] LNet: 56319:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.215448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.260836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1046.294456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1046.328612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1046.359782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.369467] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1046.370869] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1046.391298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1046.419939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1046.847732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1046.884621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1046.933453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1047.180652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1047.206281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1047.233257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1047.259276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1047.293148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1047.318762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1047.347384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1047.371360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1047.396856] LNet: 56428:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1047.422506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1047.467490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.497990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.544995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.583715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.631588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.692504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1048.223819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1048.264396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1048.296246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1048.310730] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1048.316190] LNet: Skipped 1 previous similar message [ 1048.516274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1048.551467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1048.588311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1048.638192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1048.674332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1048.708490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1048.736802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1048.769189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1048.783700] LNet: 56538:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1048.814234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1048.863495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1048.919714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1048.943289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1048.963737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1049.020063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1049.078554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1049.578187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1049.634543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1049.676462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1049.943944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1049.980440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1050.009585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1050.060459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1050.107921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1050.152083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1050.186335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1050.214299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1050.243959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1050.275449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1050.304500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1050.325672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1050.351521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1050.379553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1050.402775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1050.856977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1050.898662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1050.938833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1050.965558] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1050.977766] LNet: Skipped 2 previous similar messages [ 1051.272725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1051.305445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1051.350025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1051.420346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1051.460523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1051.501966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1051.532752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1051.569838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1051.601311] LNet: 56756:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.613938] LNet: 56756:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1051.633622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1051.664706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1051.697886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1051.747740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1051.776327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1051.818521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1051.857692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1052.454798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1052.500920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1052.556596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1052.778292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1052.811832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1052.835518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1052.859349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1052.880855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1052.905463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1052.943534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1052.984192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1053.011746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1053.057232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1053.089681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1053.119304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1053.153233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1053.181555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1053.230985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1053.521949] LNet: 56906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.531371] LNetError: 56906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.552946] LNet: Removed LNI 192.168.203.16@tcp [ 1053.936523] Key type .llcrypt unregistered [ 1053.940256] Key type ._llcrypt unregistered [ 1068.835163] Key type ._llcrypt registered [ 1068.840463] Key type .llcrypt registered [ 1070.238802] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:59:53 (1781229593) [ 1071.015480] Key type .llcrypt unregistered [ 1071.018541] Key type ._llcrypt unregistered [ 1077.805160] Key type ._llcrypt registered [ 1077.807698] Key type .llcrypt registered [ 1077.995340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.031935] alg: No test for adler32 (adler32-zlib) [ 1078.964790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1079.007347] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1086.650890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1095.798925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1095.892610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1095.930341] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1097.366535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1097.433916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 1097.488571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1097.531991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 1097.560390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1097.611269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 1097.660911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1097.707212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 1097.737416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1097.760824] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1097.769876] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1097.962160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.995324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1098.018858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1098.060420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1098.091890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1098.134641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1098.199108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1098.273746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1098.297845] LNet: 58372:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1098.321964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1098.351578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1098.389572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1098.433872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1098.474840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1098.507920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1098.565552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1099.149586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1099.194620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 1099.247331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1099.284840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 1099.313802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1099.342598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 1099.378576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1099.417339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 1099.476222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1099.492813] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1099.497119] LNet: Skipped 2 previous similar messages [ 1099.728856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1099.793868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1099.864607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1099.918325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1099.983020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1100.030404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1100.061606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1100.110710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1100.147433] LNet: 58493:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1100.176993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1100.210942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1100.242432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1100.266667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1100.305893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1100.348581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1100.386668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1101.081632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1101.150390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 1101.209859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1101.272643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 1101.341055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1101.410711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 1101.475450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1101.552648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 1101.616525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1101.634615] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1101.645401] LNet: Skipped 2 previous similar messages [ 1101.906141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1101.926408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1101.950336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1101.975954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1102.014665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1102.043760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1102.074754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1102.103332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1102.126294] LNet: 58615:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1102.147368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1102.200208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1102.240405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1102.273564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1102.331939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1102.394597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1102.460413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1103.259325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1103.309327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 1103.394328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1103.435655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 1103.480750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1103.531420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 1103.572459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1103.644594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 1103.710798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1103.748633] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1103.751870] LNet: Skipped 2 previous similar messages [ 1104.111869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.136176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1104.163809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.186342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.210493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.240730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.263624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.289584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.303292] LNet: 58736:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.314970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.341156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1104.368525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1104.428206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1104.483296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1104.544569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1104.604054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.240513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1105.288507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 1105.340590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1105.399550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 1105.469107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1105.555551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 1105.644032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1105.705850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 1105.764892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1106.066659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.114462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1106.143933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.206989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.281810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.313842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.356176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.387281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.422396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.457872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.495874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.524305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.553920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.585667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.618964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.035338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1107.108573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 1107.171554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1107.248316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 1107.300261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1107.361475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 1107.441866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1107.477926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 1107.508076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1107.761676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.782878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1107.809849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.834499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.875346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.923819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.985304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.041895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.103769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.152088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.204851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.256567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.300197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.340791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.376337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1113.182432] LNet: 59054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.200908] LNetError: 59054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.210991] LNet: Removed LNI 192.168.203.16@tcp [ 1113.231673] LNet: Removed LNI 192.168.203.16@tcp1 [ 1113.533832] Key type .llcrypt unregistered [ 1113.536024] Key type ._llcrypt unregistered [ 1127.471776] Key type ._llcrypt registered [ 1127.473421] Key type .llcrypt registered [ 1128.628383] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:00:51 (1781229651) [ 1129.585459] Key type .llcrypt unregistered [ 1129.589158] Key type ._llcrypt unregistered [ 1135.267514] Key type ._llcrypt registered [ 1135.269053] Key type .llcrypt registered [ 1135.551474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.587383] alg: No test for adler32 (adler32-zlib) [ 1136.487821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.513711] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1144.294279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1144.879922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1144.903730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1144.930858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1144.943269] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1145.146552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.171336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1145.225357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.294723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.337230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.400220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.496993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.556250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.601600] LNet: 60317:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.629988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.684496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.794547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.850645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.910959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1146.027219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1146.075654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1146.930394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1147.007097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1147.048818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1147.068642] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1147.316882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1147.374606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1147.417766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1147.458965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1147.504205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1147.565168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1147.609402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.681088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.725610] LNet: 60426:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1147.768492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.837894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.894899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.934955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.984683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1148.030814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.081408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1148.689200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1148.719522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1148.754847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1148.790301] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1149.041927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.072527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1149.101589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.122659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.157485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.220858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.250290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.282900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.303281] LNet: 60536:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.321542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.356549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.391718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.426254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.454744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.484649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1149.523735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.773712] LNet: 60577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.785047] LNetError: 60577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.809504] LNet: Removed LNI 192.168.203.16@tcp [ 1150.216959] Key type .llcrypt unregistered [ 1150.222597] Key type ._llcrypt unregistered [ 1166.892579] Key type ._llcrypt registered [ 1166.895465] Key type .llcrypt registered [ 1169.036375] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:01:31 (1781229691) [ 1170.218654] Key type .llcrypt unregistered [ 1170.226244] Key type ._llcrypt unregistered [ 1177.325291] Key type ._llcrypt registered [ 1177.328855] Key type .llcrypt registered [ 1177.568474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1177.580969] alg: No test for adler32 (adler32-zlib) [ 1178.497786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1178.534741] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1186.003553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1195.005842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1195.048839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1195.091650] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1196.460499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1196.500533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 1196.538206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1196.578535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 1196.625606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1196.666218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 1196.708808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1196.757405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 1196.806973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1196.836990] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1196.845517] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1197.080037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.111887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1197.144297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.170930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.213144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.251867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.285385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.324679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.344567] LNet: 62044:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1197.365692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.396514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1197.429828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1197.475515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1197.512694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1197.549590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1197.584982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.611857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.094068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1198.138145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 1198.187782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1198.272413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 1198.326801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1198.365371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 1198.406905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1198.446629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 1198.513837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1198.535806] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1198.547318] LNet: Skipped 2 previous similar messages [ 1198.808439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1198.856245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1198.899677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1198.947583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1198.992613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.037554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.085535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.143510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.164594] LNet: 62167:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.179855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.218234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.284566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.323801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.372611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.417186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.456090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.492357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.985321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1200.026760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 1200.055805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1200.094936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 1200.140263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1200.195331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 1200.230849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1200.276802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 1200.324592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1200.346244] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1200.619257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1200.655697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1200.724065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1200.782980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1200.844949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1200.894370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.938571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.968982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.981639] LNet: 62290:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1201.010986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.061926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.102469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.126337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.171358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.218243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.251839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1201.291614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1205.980860] LNet: 62368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.994995] LNetError: 62368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1206.010718] LNet: Removed LNI 192.168.203.16@tcp [ 1206.023266] LNet: Removed LNI 192.168.203.16@tcp1 [ 1206.426522] Key type .llcrypt unregistered [ 1206.429573] Key type ._llcrypt unregistered [ 1221.476681] Key type ._llcrypt registered [ 1221.479882] Key type .llcrypt registered [ 1222.573825] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:02:25 (1781229745) [ 1223.942337] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1225.478884] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:02:28 (1781229748) [ 1226.222453] Key type .llcrypt unregistered [ 1226.226395] Key type ._llcrypt unregistered [ 1231.326200] Key type ._llcrypt registered [ 1231.328456] Key type .llcrypt registered [ 1231.469150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1231.490905] alg: No test for adler32 (adler32-zlib) [ 1232.426322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1232.447356] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1239.606820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1240.248474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1240.284515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1240.317900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1240.360405] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1240.672980] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1240.679900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.708586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1240.737544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.776359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.809474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.847571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.898711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.932062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.950161] LNet: 63910:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.975026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.025844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.069366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.108758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.166318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.203861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.545600] LNet: 63949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.561370] LNetError: 63949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.598864] LNet: Removed LNI 192.168.203.16@tcp [ 1242.053531] Key type .llcrypt unregistered [ 1242.055897] Key type ._llcrypt unregistered [ 1258.619956] Key type ._llcrypt registered [ 1258.622970] Key type .llcrypt registered [ 1258.790223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1258.820648] alg: No test for adler32 (adler32-zlib) [ 1259.801853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1259.835583] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1266.847475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1275.166368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.218950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1275.254543] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1277.187400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1277.239883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 1277.293442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1277.318779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 1277.352494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1277.390678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 1277.478224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1277.520759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 1277.591433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1277.623391] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1277.987590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.053640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.116214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.181152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.255700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.318543] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1278.321447] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1278.326555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.329939] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1278.366598] LNet: Skipped 1 previous similar message [ 1278.367131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.428390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.462299] LNet: 64828:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.495936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.536137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.574916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.640347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.695107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.755354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.962690] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1284.323687] LNet: 64902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1284.336029] LNetError: 64902:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1284.344417] LNet: Removed LNI 192.168.203.16@tcp [ 1284.383665] LNet: Removed LNI 192.168.203.16@tcp1 [ 1284.795962] Key type .llcrypt unregistered [ 1284.799208] Key type ._llcrypt unregistered [ 1302.572323] Key type ._llcrypt registered [ 1302.574799] Key type .llcrypt registered [ 1304.102309] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:03:46 (1781229826) [ 1304.467407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1304.480445] alg: No test for adler32 (adler32-zlib) [ 1305.491453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.546790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1305.570627] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1306.447766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1306.470294] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1307.349235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1307.417693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1307.468763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1307.510295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1307.545875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1307.598442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1307.647665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1307.750908] LNetError: 66003:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1308.576366] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1308.576421] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1310.625036] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1311.456011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1314.723204] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1315.075915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1315.135518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1315.235511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1315.263321] LNet: 66110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.268859] LNetError: 66110:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.276260] LNet: Removed LNI 192.168.203.16@tcp [ 1315.286685] LNet: Removed LNI 192.168.203.16@tcp1 [ 1315.325391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.418761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1315.450933] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1316.383413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1316.421232] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1317.363498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1317.404091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.483716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1317.531502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1317.574728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1317.623792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1317.682246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1317.757964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1317.789505] LNetError: 66382:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1318.441100] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1321.248321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1324.576448] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1324.591560] LNet: Skipped 1 previous similar message [ 1329.209332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1329.268790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1329.317220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1335.999137] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:04:19 (1781229859) [ 1336.096488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.147550] LNet: 67092:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1336.163901] LNetError: 67092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.179368] LNetError: 67092:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1336.192881] LNet: Removed LNI 192.168.203.16@tcp [ 1336.230410] LNet: Removed LNI 192.168.203.16@tcp1 [ 1336.269872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1336.322426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1336.352239] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1337.204631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1337.245917] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1337.996870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1338.068200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1338.094827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1338.122162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1338.162234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1338.189852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1338.219298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 1338.314584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.337088] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.16@tcp [ 1339.360349] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1339.365080] LNet: Skipped 1 previous similar message [ 1354.162877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1354.437531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1354.514782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.629973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 500 [ 1354.768885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.680287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1361.795042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1361.869272] LNet: 67591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1361.909131] LNetError: 67591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1361.921604] LNetError: 67591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1361.941349] LNet: Removed LNI 192.168.203.16@tcp [ 1361.956728] LNet: Removed LNI 192.168.203.16@tcp1 [ 1361.997962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.089370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.125500] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1363.055301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.080893] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1364.030141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1364.066352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1364.144894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1364.190989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1364.245326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1364.280746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1364.321111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1364.349836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 1365.088590] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1365.095612] LNet: Skipped 3 previous similar messages [ 1365.682081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1375.788637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1375.816850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1375.847835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1382.741422] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:05:05 (1781229905) [ 1383.230664] LNet: 68637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1383.237781] LNetError: 68637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1383.244339] LNetError: 68637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1383.251606] LNet: Removed LNI 192.168.203.16@tcp [ 1383.265265] LNet: Removed LNI 192.168.203.16@tcp1 [ 1383.734808] Key type .llcrypt unregistered [ 1383.736669] Key type ._llcrypt unregistered [ 1389.639257] Key type ._llcrypt registered [ 1389.647497] Key type .llcrypt registered [ 1389.920940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1389.961627] alg: No test for adler32 (adler32-zlib) [ 1390.933038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1390.973399] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1399.677849] Lustre: DEBUG MARKER: Initial discovery [ 1399.798510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1402.054634] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1402.110770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1402.169887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1402.198295] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1402.234865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1402.272224] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1402.296640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -l 3 [ 1402.332785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -e local_error [ 1402.360574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m PUT -l 6 [ 1403.519130] Lustre: DEBUG MARKER: Force oleg316-client.virtnet to discover 192.168.203.116@tcp (in background) [ 1403.856886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1405.114680] Lustre: DEBUG MARKER: Force oleg316-server to discover 192.168.203.16@tcp [ 1406.944356] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1407.770960] Lustre: DEBUG MARKER: Wait for 69401 [ 1409.710209] Lustre: DEBUG MARKER: Finished wait on 69401 [ 1409.879397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1409.936097] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1409.963369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1410.252199] LNet: 69849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.259470] LNetError: 69849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1411.315204] LNet: Removed LNI 192.168.203.16@tcp [ 1411.698288] Key type .llcrypt unregistered [ 1411.702936] Key type ._llcrypt unregistered [ 1430.746762] Key type ._llcrypt registered [ 1430.750742] Key type .llcrypt registered [ 1432.200636] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:05:55 (1781229955) [ 1432.834288] Key type .llcrypt unregistered [ 1432.838711] Key type ._llcrypt unregistered [ 1439.519436] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1439.782565] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1439.800133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1439.812686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1442.476616] Key type ._llcrypt registered [ 1442.482269] Key type .llcrypt registered [ 1442.736182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1442.761988] alg: No test for adler32 (adler32-zlib) [ 1443.695731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1443.742270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1443.766664] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1444.533374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1444.554246] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1452.955795] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:06:16 (1781229976) [ 1453.153654] LNet: 71934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.160679] LNetError: 71934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.166384] LNet: Removed LNI 192.168.203.16@tcp [ 1453.185354] LNet: Removed LNI 10.1.2.3@tcp [ 1453.858897] Key type .llcrypt unregistered [ 1453.863648] Key type ._llcrypt unregistered [ 1460.585770] Key type ._llcrypt registered [ 1460.588928] Key type .llcrypt registered [ 1460.849469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1460.879279] alg: No test for adler32 (adler32-zlib) [ 1461.783704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.872137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1461.918721] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1462.607219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1462.624657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1463.690378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.16@tcp [ 1464.801268] LNetError: Unexpected error -22 connecting to 192.168.203.16@tcp at host 192.168.203.16:988 [ 1469.741783] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:06:32 (1781229992) [ 1469.808553] LNet: 73057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.828414] LNetError: 73057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.833106] LNet: Removed LNI 192.168.203.16@tcp [ 1469.839858] LNet: Removed LNI 10.1.2.3@tcp [ 1470.427435] Key type .llcrypt unregistered [ 1470.429369] Key type ._llcrypt unregistered [ 1476.036792] Key type ._llcrypt registered [ 1476.039503] Key type .llcrypt registered [ 1476.261770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1476.278614] alg: No test for adler32 (adler32-zlib) [ 1477.241580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.298793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1477.336744] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1478.001487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1478.018390] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 1478.774510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.16@tcp1 --nid 192.168.203.16@tcp2 [ 1483.077904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp2 [ 1483.096943] LNetError: 73901:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.16@tcp1 to 192.168.203.16@tcp2 - Route table may be misconfigured [ 1483.118919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp2 192.168.203.16@tcp1 [ 1489.510692] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:06:52 (1781230012) [ 1489.756173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.790099] LNet: 74474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.808197] LNetError: 74474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.812577] LNet: Removed LNI 192.168.203.16@tcp1 [ 1490.857375] LNet: Removed LNI 192.168.203.16@tcp2 [ 1490.912584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.019442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.052130] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1491.752078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.780712] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1492.853081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1492.904564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.16@tcp [ 1492.988641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp1 [ 1493.046416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp -e network_timeout [ 1493.092743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -e network_timeout [ 1493.143906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -e network_timeout [ 1493.233099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -e network_timeout [ 1493.279888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1493.299650] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1493.318576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1493.445561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1493.589718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1494.050067] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1499.899657] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:07:03 (1781230023) [ 1500.101687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.146990] LNet: 75335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.168891] LNetError: 75335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.179116] LNetError: 75335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1500.187456] LNet: Removed LNI 192.168.203.16@tcp [ 1501.225947] LNet: Removed LNI 192.168.203.16@tcp1 [ 1501.253828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.349557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1501.678624] LNet: 75374:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1502.034909] Key type .llcrypt unregistered [ 1502.037757] Key type ._llcrypt unregistered [ 1511.748987] Key type ._llcrypt registered [ 1511.751525] Key type .llcrypt registered [ 1512.872459] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:07:16 (1781230036) [ 1513.217374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1513.239920] alg: No test for adler32 (adler32-zlib) [ 1514.166170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.480553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1517.008753] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:07:20 (1781230040) [ 1517.084725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.101851] LNet: 76465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.128301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.184441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.205670] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1517.888484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1517.926400] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1518.691811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1518.729863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp1 [ 1518.755357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp1 [ 1524.064980] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:07:27 (1781230047) [ 1525.628340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg316-server [ 1527.169084] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:07:30 (1781230050) [ 1528.852241] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg316-server [ 1530.456916] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:07:33 (1781230053) [ 1532.372890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg316-server [ 1533.949825] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:07:37 (1781230057) [ 1535.632302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg316-server [ 1537.487818] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:07:40 (1781230060) [ 1539.214708] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg316-server [ 1540.802341] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:07:44 (1781230064) [ 1542.641824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg316-server [ 1543.949941] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:07:47 (1781230067) [ 1545.930349] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg316-server [ 1547.362443] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:07:50 (1781230070) [ 1549.331519] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg316-server [ 1551.050631] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:07:54 (1781230074) [ 1552.918786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg316-server [ 1554.556571] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:07:57 (1781230077) [ 1554.669926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.696282] LNet: 80347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.706494] LNetError: 80347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.745769] LNet: Removed LNI 192.168.203.16@tcp [ 1555.751949] LNetError: 80347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.785464] LNet: Removed LNI 192.168.203.16@tcp1 [ 1555.821117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.919560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.945418] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1556.702674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1556.772573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1556.951895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.972263] LNet: 80495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.982293] LNetError: 80495:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.059319] LNet: Removed LNI 192.168.203.16@tcp [ 1558.084865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.134223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1558.162042] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1558.934905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1559.106913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.135713] LNet: 80641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1559.143994] LNetError: 80641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1559.166667] LNet: Removed LNI 192.168.203.16@tcp [ 1559.189833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.254439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1559.275906] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1560.004378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1560.158666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1560.444682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.510814] LNet: Removed LNI 192.168.203.16@tcp [ 1561.536800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.598356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1561.614173] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1562.291387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1562.425739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.500609] LNet: Removed LNI 192.168.203.16@tcp [ 1563.542448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.604951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.637972] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1564.357109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1564.437834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1564.605154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.643567] LNet: 81081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.652169] LNet: 81081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1564.661889] LNetError: 81081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.672201] LNetError: 81081:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1564.696805] LNet: Removed LNI 192.168.203.16@tcp [ 1564.748864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.813862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1564.846543] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1565.608985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1567.003564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.039778] LNet: Removed LNI 192.168.203.16@tcp [ 1568.094456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.150405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1568.192557] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1568.812771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1568.866674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1569.083451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.153726] LNet: Removed LNI 192.168.203.16@tcp [ 1570.190353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.262905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1570.317368] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1570.986351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1572.394575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.478834] LNet: Removed LNI 192.168.203.16@tcp [ 1573.541485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.646357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.665759] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1574.295673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1574.380902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1575.845919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.870966] LNet: 81693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.876921] LNet: 81693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1575.888906] LNetError: 81693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.895766] LNetError: 81693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1575.904510] LNet: Removed LNI 192.168.203.16@tcp [ 1575.936983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.015220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1576.048572] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1576.754447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1576.993735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.088895] LNet: Removed LNI 192.168.203.16@tcp [ 1578.127983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.204458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.230629] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1578.852674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1578.926049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1579.229655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.264480] LNet: Removed LNI 192.168.203.16@tcp [ 1580.288749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.347849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1580.375917] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1581.108728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1581.445086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.570495] LNet: Removed LNI 192.168.203.16@tcp [ 1582.609401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.703037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.732144] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1583.364718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1583.433441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.465816] LNet: Removed LNI 192.168.203.16@tcp [ 1583.491373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.606503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1583.630761] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1584.390197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1591.586083] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:08:34 (1781230114) [ 1591.715357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.799177] LNet: Removed LNI 192.168.203.16@tcp [ 1591.835823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.950479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1591.993864] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1593.508502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.533164] LNet: 83106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.537990] LNet: 83106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1593.544481] LNetError: 83106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.557838] LNetError: 83106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1593.581279] LNet: Removed LNI 192.168.203.16@tcp231 [ 1593.596660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.647467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1593.669152] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1593.673317] LNet: 83114:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1594.062599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1594.094842] LNet: Removed LNI 192.168.203.16@tcp231 [ 1594.110304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1594.131942] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1595.333966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.370219] LNet: Removed LNI 192.168.203.16@tcp231 [ 1595.387341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.470976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1595.508567] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 1595.511959] LNet: 83266:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1596.097221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1596.141377] LNet: Removed LNI 192.168.203.16@tcp231 [ 1596.157197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1596.172945] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 1597.157745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.194348] LNet: Removed LNI 192.168.203.16@tcp231 [ 1597.220937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.292646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1597.312139] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 1597.318241] LNet: 83417:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1597.805295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1597.836186] LNet: Removed LNI 192.168.203.16@tcp231 [ 1597.864457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1597.896506] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 1599.175909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.231093] LNet: Removed LNI 192.168.203.16@tcp231 [ 1599.256816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.335685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1599.362767] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1599.369632] LNet: 83569:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1599.826774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1599.871787] LNet: Removed LNI 192.168.203.16@tcp231 [ 1599.900231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1599.942905] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1607.973900] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:08:50 (1781230130) [ 1608.083738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.129274] LNet: Removed LNI 192.168.203.16@tcp231 [ 1608.157350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.270964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1608.304651] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1609.136508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.178790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.215686] LNet: Removed LNI 192.168.203.16@tcp232 [ 1609.243433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1609.280863] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1610.067810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1610.107208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1610.146225] LNet: Removed LNI 192.168.203.16@tcp232 [ 1610.174575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1610.219568] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1610.989818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1611.032680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1611.074303] LNet: Removed LNI 192.168.203.16@tcp232 [ 1611.096475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1611.146325] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1611.887307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1611.916451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1611.941683] LNet: Removed LNI 192.168.203.16@tcp232 [ 1611.958079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1617.769954] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:09:00 (1781230140) [ 1618.769112] Key type .llcrypt unregistered [ 1618.776300] Key type ._llcrypt unregistered [ 1625.041582] Key type ._llcrypt registered [ 1625.049578] Key type .llcrypt registered [ 1625.300560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.316896] alg: No test for adler32 (adler32-zlib) [ 1626.237444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1626.257938] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1633.076393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1641.992703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.019438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1642.046534] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1643.525633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 1643.564584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 1643.618138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 1643.673426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 1643.727123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 1643.783668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 1643.845160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 1643.899760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 1643.950318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1643.990206] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1644.000934] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1644.277245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1644.318918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1644.362828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1644.388323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1644.411841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1644.436938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1644.477701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1644.502521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1644.519561] LNet: 86068:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1644.536167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1644.558023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1644.596569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1644.627332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1644.651196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1644.670645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1645.075948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 1645.101656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 1645.134838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 1645.171249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 1645.208202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 1645.240904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 1645.273950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 1645.301744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 1645.341498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1645.381534] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1645.595443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1645.625155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1645.654850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1645.677803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1645.700708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1645.738958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1645.774659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1645.806355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1645.816735] LNet: 86188:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1645.830804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1645.870462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1645.911688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1645.971952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1646.008532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1646.066430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1646.672961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 1646.719971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 1646.755192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 1646.795760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 1646.829292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 1646.863833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 1646.905574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 1646.932957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 1646.956106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1646.969501] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1647.165800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1647.186879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1647.211473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1647.235722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1647.262740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1647.284546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1647.324483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1647.363841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1647.376602] LNet: 86309:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1647.399760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1647.439571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1647.479755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1647.505350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1647.535492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1647.574135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1648.184469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 1648.220416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 1648.256468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 1648.307859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 1648.345453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 1648.379875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 1648.410751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 1648.481759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 1648.529557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1648.849965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1648.878553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1648.894391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.908519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.928141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.972154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1649.011135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1649.042213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1649.086754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1649.121323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1649.164063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1649.214355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1649.263938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1649.312985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.928972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 1649.964958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 1649.999656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 1650.038291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 1650.097167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 1650.138311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 1650.194230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 1650.246905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 1650.293324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1650.330792] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1650.335436] LNet: Skipped 1 previous similar message [ 1650.569581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1650.595914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1650.616113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1650.637207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.659767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.681951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.707876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.737543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.755506] LNet: 86549:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.763396] LNet: 86549:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1650.785033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.832204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.886254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.923426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.955682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.980528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1651.565931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 1651.608659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 1651.674990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 1651.748983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 1651.796289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 1651.832456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 1651.862871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 1651.901280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 1651.944842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1651.963542] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1652.216908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1652.268134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1652.325987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1652.382923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1652.421573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1652.465314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1652.506963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.555399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1652.606187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1652.653406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1652.678164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1652.700185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1652.727425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1652.762342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.897440] LNet: 86743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.933259] LNetError: 86743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.940780] LNet: Removed LNI 192.168.203.16@tcp [ 1658.950258] LNet: Removed LNI 192.168.203.16@tcp1 [ 1659.392959] Key type .llcrypt unregistered [ 1659.395727] Key type ._llcrypt unregistered [ 1675.075630] Key type ._llcrypt registered [ 1675.078540] Key type .llcrypt registered [ 1676.457832] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:09:59 (1781230199) [ 1676.827290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.854673] alg: No test for adler32 (adler32-zlib) [ 1677.800712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.850714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.868088] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1678.537734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1678.563625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1678.579609] LNetError: 87721:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1678.656692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.16@tcp [ 1678.765100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1684.248115] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:10:07 (1781230207) [ 1684.441720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.492705] LNet: 88310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.506807] LNetError: 88310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.535666] LNet: Removed LNI 192.168.203.16@tcp [ 1684.576976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.648066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.689603] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1685.361986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1685.385978] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1686.302555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.16@tcp [ 1686.332822] LNetError: 88577:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1686.689940] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1686.713263] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1687.687723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.16@tcp [ 1694.480412] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:10:17 (1781230217) [ 1696.002799] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg316-server [ 1697.418214] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:10:20 (1781230220) [ 1698.945085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg316-server [ 1700.222595] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:10:23 (1781230223) [ 1702.008847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg316-server [ 1704.044314] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:10:26 (1781230226) [ 1704.169278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.199225] LNet: 90196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.209507] LNetError: 90196:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.215369] LNet: Removed LNI 192.168.203.16@tcp [ 1704.230686] LNet: Removed LNI 192.168.203.16@tcp1 [ 1704.284475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.333790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.355907] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1705.078789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1705.156421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1705.211576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1705.334504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1705.394780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1705.431234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1711.045268] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:10:34 (1781230234) [ 1711.163801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.189165] LNet: 90922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.195238] LNetError: 90922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1711.201757] LNetError: 90922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1711.212058] LNet: Removed LNI 192.168.203.16@tcp [ 1711.245845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.285425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.302850] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1718.173376] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:10:41 (1781230241) [ 1718.283379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.311570] LNet: 91623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.325487] LNetError: 91623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.351896] LNet: Removed LNI 192.168.203.16@tcp [ 1718.383034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.459359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.483863] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1724.799911] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:10:48 (1781230248) [ 1726.597726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1728.425957] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:10:51 (1781230251) [ 1728.893636] LNet: 92663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.899265] LNetError: 92663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.918179] LNet: Removed LNI 192.168.203.16@tcp [ 1729.376243] Key type .llcrypt unregistered [ 1729.382277] Key type ._llcrypt unregistered [ 1735.394706] Key type ._llcrypt registered [ 1735.397174] Key type .llcrypt registered [ 1735.647258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1735.681144] alg: No test for adler32 (adler32-zlib) [ 1736.678716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1736.718854] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1743.737277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1749.669344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.116@tcp [ 1750.105758] LNet: 93080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.113289] LNetError: 93080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.145330] LNet: Removed LNI 192.168.203.16@tcp [ 1750.574442] Key type .llcrypt unregistered [ 1750.580585] Key type ._llcrypt unregistered [ 1761.339580] Key type ._llcrypt registered [ 1761.341732] Key type .llcrypt registered [ 1762.452687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1764.257339] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1765.717384] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:11:29 (1781230289) [ 1767.141885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1768.709404] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:11:31 (1781230291) [ 1770.351412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg316-server [ 1771.823137] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:11:35 (1781230295) [ 1773.647774] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg316-server [ 1775.161391] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:11:38 (1781230298) [ 1776.460392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.591969] alg: No test for adler32 (adler32-zlib) [ 1777.681244] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 1777.945198] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1779.651609] Key type lgssc registered [ 1781.062588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1799.473315] Key type lgssc unregistered [ 1799.737843] LNet: 96262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.742572] LNetError: 96262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.763317] LNet: Removed LNI 192.168.203.16@tcp [ 1800.451429] Key type .llcrypt unregistered [ 1800.453121] Key type ._llcrypt unregistered [ 1812.533976] Key type ._llcrypt registered [ 1812.537646] Key type .llcrypt registered [ 1814.060691] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:12:16 (1781230336) [ 1814.535520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1814.545302] alg: No test for adler32 (adler32-zlib) [ 1815.499532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.598520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1815.654573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1815.726964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1815.781428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1821.981077] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:12:25 (1781230345) [ 1822.140186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.179831] LNet: 97671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.214565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.319846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.349053] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1823.386287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.425811] LNet: 97819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.433763] LNetError: 97819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.453738] LNet: Removed LNI 192.168.203.16@tcp [ 1823.480786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.544806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1823.574736] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1824.043771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.095822] LNetError: 97847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.111909] LNet: Removed LNI 192.168.203.16@tcp [ 1824.149711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.216475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1824.242694] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1830.462490] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:12:33 (1781230353) [ 1830.753613] LNet: 98438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1830.758289] LNet: 98438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1830.764447] LNetError: 98438:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1830.783583] LNet: Removed LNI 192.168.203.16@tcp [ 1831.421254] Key type .llcrypt unregistered [ 1831.424475] Key type ._llcrypt unregistered [ 1838.249211] Key type ._llcrypt registered [ 1838.253255] Key type .llcrypt registered [ 1838.466299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1838.479574] alg: No test for adler32 (adler32-zlib) [ 1839.441720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.494321] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1839.499968] LNet: *** cfs_fail_loc=e003, val=100*** [ 1840.247357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1840.266453] LNet: *** cfs_fail_loc=e003, val=100*** [ 1840.269860] LNet: Skipped 2 previous similar messages [ 1840.284881] Ping reply received with mapping: [ 1840.290343] found metadata reply NI[192.168.203.16@tcp] [ 1840.293873] 10 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1840.297178] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1840.302887] 5a 5a 5a 5a 5a 5a 5a 5a [ 1840.417785] LNet: 98836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.433083] LNetError: 98836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.452077] LNet: Removed LNI 192.168.203.16@tcp [ 1841.084368] Key type .llcrypt unregistered [ 1841.086464] Key type ._llcrypt unregistered [ 1852.750774] Key type ._llcrypt registered [ 1852.752793] Key type .llcrypt registered [ 1854.084506] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:12:57 (1781230377) [ 1854.579267] Key type .llcrypt unregistered [ 1854.581210] Key type ._llcrypt unregistered [ 1860.779846] Key type ._llcrypt registered [ 1860.782498] Key type .llcrypt registered [ 1860.914865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1860.928821] alg: No test for adler32 (adler32-zlib) [ 1862.047085] LNetError: 99890:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1862.472401] Key type .llcrypt unregistered [ 1862.475758] Key type ._llcrypt unregistered [ 1867.340768] Key type ._llcrypt registered [ 1867.343288] Key type .llcrypt registered [ 1868.664039] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:13:11 (1781230391) [ 1869.250949] Key type .llcrypt unregistered [ 1869.255346] Key type ._llcrypt unregistered [ 1875.872583] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1875.886421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1878.359456] Key type ._llcrypt registered [ 1878.362084] Key type .llcrypt registered [ 1878.541987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1878.556612] alg: No test for adler32 (adler32-zlib) [ 1879.532426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1879.616177] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1889.734566] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:13:32 (1781230412) [ 1889.846541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.881333] LNet: 101664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.897558] LNetError: 101664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1889.918642] LNet: Removed LNI 10.1.2.3@tcp [ 1889.956464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1890.878620] Key type .llcrypt unregistered [ 1890.887895] Key type ._llcrypt unregistered [ 1897.606712] Key type ._llcrypt registered [ 1897.610565] Key type .llcrypt registered [ 1897.816946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1897.842218] alg: No test for adler32 (adler32-zlib) [ 1898.864410] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1899.006583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1899.128682] LNet: 101958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.136440] LNetError: 101958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.149527] LNet: Removed LNI 192.168.203.16@tcp [ 1899.911525] Key type .llcrypt unregistered [ 1899.917163] Key type ._llcrypt unregistered [ 1912.039282] Key type ._llcrypt registered [ 1912.043134] Key type .llcrypt registered [ 1913.444649] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:13:56 (1781230436) [ 1914.087557] Key type .llcrypt unregistered [ 1914.092152] Key type ._llcrypt unregistered [ 1920.274629] Key type ._llcrypt registered [ 1920.279210] Key type .llcrypt registered [ 1920.627542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1920.657470] alg: No test for adler32 (adler32-zlib) [ 1928.201037] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:14:11 (1781230451) [ 1928.376497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.413774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.520886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.542063] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1929.479807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.521762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1929.566855] LNet: Removed LNI 192.168.203.16@tcp [ 1929.850418] LNet: 103703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.293553] Key type .llcrypt unregistered [ 1930.301224] Key type ._llcrypt unregistered [ 1942.665951] Key type ._llcrypt registered [ 1942.667595] Key type .llcrypt registered [ 1944.131473] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:14:26 (1781230466) [ 1944.621389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.638803] alg: No test for adler32 (adler32-zlib) [ 1945.646909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.733242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1945.780868] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1946.702676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1946.765893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.16@tcp [ 1946.801220] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1946.804725] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.808920] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 0 [ 1946.817985] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1946.825137] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1946.836411] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.841921] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 0 [ 1946.847595] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1946.850655] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1946.854236] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.861234] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 1996 [ 1946.869687] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1946.872496] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1946.878043] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.885479] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 1996 [ 1946.893708] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1954.335486] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:14:37 (1781230477) [ 1954.996379] LNet: 105242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.015758] LNetError: 105242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1955.037683] LNet: Removed LNI 192.168.203.16@tcp [ 1955.844461] Key type .llcrypt unregistered [ 1955.848294] Key type ._llcrypt unregistered [ 1962.460921] Key type ._llcrypt registered [ 1962.468571] Key type .llcrypt registered [ 1962.897748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1962.926075] alg: No test for adler32 (adler32-zlib) [ 1963.940328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1963.988781] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1972.179686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1982.801446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1982.899394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1982.944334] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1984.539532] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1984.662783] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1984.711854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1984.724657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1987.268069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1987.291775] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1998.168417] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:15:21 (1781230521) [ 1998.390337] LNet: 106754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.399243] LNetError: 106754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.406599] LNet: Removed LNI 192.168.203.16@tcp [ 1999.456200] LNet: Removed LNI 192.168.203.16@tcp1 [ 1999.460801] LNetError: 106754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1999.477325] LNetError: 106754:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1999.502064] LNet: Removed LNI 10.1.2.3@tcp99 [ 2000.159668] Key type .llcrypt unregistered [ 2000.162495] Key type ._llcrypt unregistered [ 2006.065297] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2006.071148] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2008.592820] Key type ._llcrypt registered [ 2008.597020] Key type .llcrypt registered [ 2008.763473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.787355] alg: No test for adler32 (adler32-zlib) [ 2009.703762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.769907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2009.807120] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2010.513680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2010.540764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2012.521887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2012.576426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2012.610098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2012.627558] LNetError: 107499:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2012.658685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2012.718254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2013.349274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2013.433686] LNet: 107519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.439836] LNetError: 107519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2013.447508] LNet: Removed LNI 192.168.203.16@tcp [ 2013.472542] LNet: Removed LNI 10.1.2.3@tcp [ 2014.153442] Key type .llcrypt unregistered [ 2014.155056] Key type ._llcrypt unregistered [ 2025.037536] Key type ._llcrypt registered [ 2025.039319] Key type .llcrypt registered [ 2026.363680] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:15:49 (1781230549) [ 2026.788859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2026.801603] alg: No test for adler32 (adler32-zlib) [ 2027.687107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2027.731310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2027.747184] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2028.472086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2028.569308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg316-client@tcp [ 2034.763788] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:15:57 (1781230557) [ 2034.911726] LNet: 109050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.926632] LNetError: 109050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.944828] LNet: Removed LNI 192.168.203.16@tcp [ 2036.710372] Key type .llcrypt unregistered [ 2036.714443] Key type ._llcrypt unregistered [ 2043.549800] Key type ._llcrypt registered [ 2043.554094] Key type .llcrypt registered [ 2043.748965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2043.764409] alg: No test for adler32 (adler32-zlib) [ 2044.703602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2044.743482] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2044.790575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2050.688326] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:16:14 (1781230574) [ 2052.246656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2053.714735] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:16:16 (1781230576) [ 2054.985452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2056.277662] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:16:19 (1781230579) [ 2056.366987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2056.399041] LNet: 110468:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.405769] LNetError: 110468:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2056.426745] LNet: Removed LNI 192.168.203.16@tcp [ 2056.454732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.514923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2056.549620] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2057.264867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2057.318160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2063.579277] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:16:26 (1781230586) [ 2063.757950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.771255] LNet: 111184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.774556] LNetError: 111184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.801500] LNet: Removed LNI 192.168.203.16@tcp [ 2063.818540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.852463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2063.873599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2064.264942] Key type .llcrypt unregistered [ 2064.266767] Key type ._llcrypt unregistered [ 2076.406461] Key type ._llcrypt registered [ 2076.408606] Key type .llcrypt registered [ 2077.881938] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:16:40 (1781230600) [ 2078.283950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2078.297025] alg: No test for adler32 (adler32-zlib) [ 2079.256556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.337367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.368457] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2080.571307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2080.703319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2088.828590] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:16:51 (1781230611) [ 2088.918798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.941947] LNet: 112733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.956512] LNetError: 112733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.036636] LNet: Removed LNI 192.168.203.16@tcp [ 2090.082606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.161891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2090.211831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2096.577310] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:16:59 (1781230619) [ 2096.692478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.719222] LNet: 113312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.747945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.838080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2097.453388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2097.488890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2097.577543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2097.904749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2097.941860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2098.011836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.026762] LNet: 113353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.040164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.080048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2098.349340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2098.376521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2098.418055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2098.805618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2098.868198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2105.944354] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:17:09 (1781230629) [ 2106.125430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.150469] LNet: 113953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.167514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.226333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2106.262527] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2107.012287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2107.031665] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2108.052667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2108.118639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2108.392462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.418177] LNetError: 114232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.472451] LNet: Removed LNI 192.168.203.16@tcp [ 2109.479443] LNetError: 114232:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2109.499414] LNet: Removed LNI 192.168.203.16@tcp2 [ 2109.556695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.630928] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2109.639587] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2109.645583] LNet: 114238:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2110.239703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2110.539602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2110.572626] LNet: 114276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.585704] LNet: 114276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2111.648143] LNet: Removed LNI 192.168.203.16@tcp [ 2111.652525] LNetError: 114276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2111.671098] LNetError: 114276:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2111.697715] LNet: Removed LNI 192.168.203.16@tcp2 [ 2111.773578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.885266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2111.963263] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2112.949227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2112.978125] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2113.748426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2113.792692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2113.843245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2114.027134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.104209] LNet: Removed LNI 192.168.203.16@tcp [ 2116.135971] LNet: Removed LNI 192.168.203.16@tcp2 [ 2116.163992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.236750] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2116.253870] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2116.265042] LNet: 114566:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2116.794676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2116.969614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.990379] LNetError: 114604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.997273] LNetError: 114604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2117.003124] LNet: Removed LNI 192.168.203.16@tcp [ 2118.061740] LNet: Removed LNI 192.168.203.16@tcp2 [ 2118.104826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.191366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2118.229878] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2118.968645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2119.006572] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2119.967971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2120.049520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2120.270339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2120.294908] LNet: 114883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.300730] LNet: 114883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2121.313446] LNet: Removed LNI 192.168.203.16@tcp [ 2121.329640] LNet: Removed LNI 192.168.203.16@tcp2 [ 2121.369504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.448935] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2121.456741] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2121.468928] LNet: 114889:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2122.140075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2122.388351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.488154] LNet: Removed LNI 192.168.203.16@tcp [ 2123.504402] LNet: Removed LNI 192.168.203.16@tcp2 [ 2123.540917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.628741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.659256] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2124.441876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2124.465291] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2125.401895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2125.443540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2125.512137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2125.719696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.744956] LNetError: 115212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.750276] LNetError: 115212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2125.756768] LNet: Removed LNI 192.168.203.16@tcp [ 2126.823331] LNet: Removed LNI 192.168.203.16@tcp2 [ 2126.840921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.915571] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2126.926465] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2126.937336] LNet: 115218:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2127.476108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2127.687747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.708669] LNet: Removed LNI 192.168.203.16@tcp [ 2128.742914] LNet: Removed LNI 192.168.203.16@tcp2 [ 2128.810798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.904919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2128.929452] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2129.689775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.715705] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2130.588592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2130.630462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2130.691272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2131.024992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.064155] LNet: Removed LNI 192.168.203.16@tcp [ 2132.075061] LNet: Removed LNI 192.168.203.16@tcp2 [ 2132.145888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.297537] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2132.319340] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2132.324283] LNet: 115558:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.760025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2133.184786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.240194] LNet: Removed LNI 192.168.203.16@tcp [ 2134.253713] LNet: Removed LNI 192.168.203.16@tcp2 [ 2134.322134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.366833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.416983] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2135.384831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.409609] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2136.179362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2136.205778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2136.245844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2136.312674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2136.726378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.751104] LNet: 115911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.755870] LNet: 115911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2137.824498] LNet: Removed LNI 192.168.203.16@tcp [ 2138.863642] LNet: Removed LNI 192.168.203.16@tcp2 [ 2138.945161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.056504] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2139.063373] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2139.726791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2146.678343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2148.294133] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:17:51 (1781230671) [ 2148.421831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2148.461481] LNetError: 116673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.471287] LNetError: 116673:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2148.478750] LNet: Removed LNI 192.168.203.16@tcp [ 2149.544118] LNet: Removed LNI 192.168.203.16@tcp2 [ 2149.575843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.915872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2149.946533] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2150.523195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2150.540567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2151.430933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.452711] LNet: Removed LNI 192.168.203.16@tcp [ 2151.463978] LNet: Removed LNI 10.1.2.3@tcp [ 2151.487498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.576382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2151.607562] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2151.611272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2152.126242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2152.152208] LNet: Removed LNI 192.168.203.16@tcp [ 2152.164070] LNet: Removed LNI 10.1.2.3@tcp [ 2158.096890] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:18:01 (1781230681) [ 2158.324167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2158.397523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.579706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2158.706539] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2158.711084] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2158.716209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2161.010280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.055176] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2161.824533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2161.836314] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2164.573542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2164.641102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2164.719866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.16@tcp -m PUT -e local_timeout -r 1 [ 2164.769262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2164.853070] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2164.861769] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2165.098901] LNetError: 117618:0:(lib-move.c:4710:lnet_parse()) 192.168.203.16@tcp, src 192.168.203.16@tcp: Dropping PUT (error -108 looking up sender) [ 2167.137331] LNet: Removed LNI 192.168.203.16@tcp [ 2167.153272] LNet: Removed LNI 10.1.2.3@tcp [ 2167.553125] Key type .llcrypt unregistered [ 2167.557618] Key type ._llcrypt unregistered [ 2178.951477] Key type ._llcrypt registered [ 2178.954312] Key type .llcrypt registered [ 2180.090313] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:18:23 (1781230703) [ 2180.500853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.515197] alg: No test for adler32 (adler32-zlib) [ 2181.418758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.853786] LNet: 118929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.487470] Key type .llcrypt unregistered [ 2182.494079] Key type ._llcrypt unregistered [ 2188.921679] Key type ._llcrypt registered [ 2188.926918] Key type .llcrypt registered [ 2189.130233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2189.157787] alg: No test for adler32 (adler32-zlib) [ 2190.134309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2190.165501] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2197.896426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2198.110110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2198.804712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 2199.176988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2199.215640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2199.265336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2199.319683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2199.348127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2199.384815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2199.392350] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 2199.440264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2199.485052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2199.508726] LNet: 119380:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2199.532306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2199.570969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2199.598323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2199.633922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2199.682635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2199.722237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2200.306455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2200.518763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2200.551182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2200.611637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2200.642371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2200.663559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2200.688445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2200.711955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2200.742478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2200.758782] LNet: 119497:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2200.774147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2200.798666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2200.821647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2200.871479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2200.898448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2200.948561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2201.591584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 2201.807846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2201.837262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2201.863882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2201.910776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2201.958564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2201.985128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2202.018747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2202.042763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2202.070632] LNet: 119614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2202.097491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2202.138938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2202.165827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2202.203890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2202.267545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2202.314092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2202.962707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2203.169951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2203.237372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2203.279427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2203.327755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2203.365093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2203.417798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2203.475777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2203.527425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2203.574621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2203.619320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2203.678874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2203.738516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2203.786820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2203.834763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.268025] LNet: 119778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.280934] LNetError: 119778:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2204.299539] LNet: Removed LNI 192.168.203.16@tcp [ 2204.681950] Key type .llcrypt unregistered [ 2204.687911] Key type ._llcrypt unregistered [ 2218.320350] Key type ._llcrypt registered [ 2218.321890] Key type .llcrypt registered [ 2219.685576] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:19:02 (1781230742) [ 2220.099759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.120413] alg: No test for adler32 (adler32-zlib) [ 2221.051249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.507578] LNet: 120664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.108877] Key type .llcrypt unregistered [ 2222.117113] Key type ._llcrypt unregistered [ 2228.250306] Key type ._llcrypt registered [ 2228.253975] Key type .llcrypt registered [ 2228.498917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2228.519250] alg: No test for adler32 (adler32-zlib) [ 2229.521060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2229.559634] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2237.330437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2245.962881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2246.033074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2246.066505] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 2247.209612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2247.814312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 2248.032203] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 2248.059178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2248.099325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2248.123661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2248.155888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2248.185393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2248.209556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2248.236369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2248.269875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2248.288438] LNet: 121316:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2248.307558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2248.349536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2248.378217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2248.410420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2248.435401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2248.472883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2249.095345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2249.321106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2249.344211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2249.377560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.412635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.444482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2249.476466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2249.505724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2249.555805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2249.593634] LNet: 121433:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2249.636970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2249.687959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2249.725841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2249.766656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2249.803774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2249.840396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2250.524947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 2250.793814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2250.825715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2250.857535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2250.884472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2250.912553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2250.941573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2250.972706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2250.997653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2251.017664] LNet: 121550:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.034952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2251.071474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2251.101938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2251.133916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2251.172318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2251.209339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2251.815360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2252.015685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.044939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2252.066222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.089655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.109670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.134780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.172826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2252.211173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2252.258443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2252.309946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2252.335688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2252.369391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2252.403319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2252.430441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2256.892590] LNet: 121748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2256.899268] LNetError: 121748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.952207] LNet: Removed LNI 192.168.203.16@tcp [ 2257.954991] LNetError: 121748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2257.980402] LNet: Removed LNI 192.168.203.16@tcp1 [ 2258.315816] Key type .llcrypt unregistered [ 2258.319431] Key type ._llcrypt unregistered [ 2271.711705] Key type ._llcrypt registered [ 2271.718505] Key type .llcrypt registered [ 2272.990161] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:19:56 (1781230796) [ 2279.147685] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:20:02 (1781230802) [ 2285.635351] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:20:08 (1781230808) [ 2293.489150] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:20:16 (1781230816) [ 2301.855099] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:20:24 (1781230824) [ 2309.641726] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:20:32 (1781230832) [ 2317.177281] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:20:40 (1781230840) [ 2325.441826] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:20:48 (1781230848) [ 2333.214428] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:20:56 (1781230856) [ 2340.802495] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:21:03 (1781230863) [ 2347.661844] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:21:10 (1781230870) [ 2354.922118] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:21:18 (1781230878) [ 2362.048541] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:21:25 (1781230885) [ 2367.973878] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:21:31 (1781230891) [ 2374.760992] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:21:37 (1781230897) [ 2381.583056] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:21:44 (1781230904) [ 2388.925493] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:21:52 (1781230912) [ 2396.287563] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:21:59 (1781230919) [ 2396.661932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2396.674503] alg: No test for adler32 (adler32-zlib) [ 2397.617689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.757245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2397.786134] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2398.951274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.975559] LNet: 132458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.981749] LNetError: 132458:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.003266] LNet: Removed LNI 192.168.203.16@tcp [ 2399.045312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.123319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2399.146597] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2406.043286] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:22:09 (1781230929) [ 2413.795960] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:22:16 (1781230936) [ 2421.139851] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:22:24 (1781230944) [ 2428.973248] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:22:32 (1781230952) [ 2436.698317] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:22:39 (1781230959) [ 2444.476915] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:22:47 (1781230967) [ 2452.236328] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:22:55 (1781230975) [ 2459.208465] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:23:02 (1781230982) [ 2467.444211] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:23:10 (1781230990) [ 2475.327338] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:23:18 (1781230998) [ 2482.719657] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:23:25 (1781231005) [ 2489.559285] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:23:32 (1781231012) [ 2496.436509] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2225 sec ========== 22:23:39 (1781231019) [ 2496.734145] LNet: 139925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.746405] LNetError: 139925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.769606] LNet: Removed LNI 192.168.203.16@tcp [ 2497.533626] Key type .llcrypt unregistered [ 2497.536253] Key type ._llcrypt unregistered [ 2509.581164] Key type ._llcrypt registered [ 2509.583527] Key type .llcrypt registered [ 2509.893562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2509.923739] alg: No test for adler32 (adler32-zlib) [ 2510.993736] Lustre: Lustre: Build Version: 2.17.53_64_gab2e0dd [ 2511.326490] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2513.032154] Key type lgssc registered [ 2514.151950] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.243431] Lustre: Mounted lustre-client [ 2587.525638] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2597.770086] Lustre: Unmounted lustre-client [ 2619.896302] Key type lgssc unregistered [ 2620.285694] LNet: 142851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2620.303049] LNetError: 142851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.336920] LNet: Removed LNI 192.168.203.16@tcp [ 2621.556186] Key type .llcrypt unregistered [ 2621.559199] Key type ._llcrypt unregistered