[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466849897 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524576K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002324] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007456] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010139] LSM: Security Framework initializing [ 0.011057] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013075] *** VALIDATE selinux *** [ 0.020719] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024795] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027136] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] *** VALIDATE tmpfs *** [ 0.030059] *** VALIDATE proc *** [ 0.031237] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033357] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036034] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040600] debug: unmapping init [mem 0xffffffffa8859000-0xffffffffa8860fff] [ 0.042185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043789] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.046013] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049008] ... fixed-purpose events: 3 [ 0.049866] ... event mask: 000000070000000f [ 0.050277] rcu: Hierarchical SRCU implementation. [ 0.052412] smp: Bringing up secondary CPUs ... [ 0.053598] x86: Booting SMP configuration: [ 0.054025] .... node #0, CPUs: #1 #2 #3 [ 0.057194] smp: Brought up 1 node, 4 CPUs [ 0.058985] smpboot: Max logical packages: 1 [ 0.059011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135369] node 0 deferred pages initialised in 73ms [ 0.139033] devtmpfs: initialized [ 0.140243] x86/mm: Memory block size: 128MB [ 0.142618] gcov: version magic: 0x41383552 [ 0.144209] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146286] pinctrl core: initialized pinctrl subsystem [ 0.147145] [ 0.147475] ************************************************************* [ 0.148008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149009] ** ** [ 0.150011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.151009] ** ** [ 0.152009] ** This means that this kernel is built to expose internal ** [ 0.153011] ** IOMMU data structures, which may compromise security on ** [ 0.154011] ** your system. ** [ 0.155010] ** ** [ 0.156010] ** If you see this message and you are not debugging the ** [ 0.157010] ** kernel, report this immediately to your vendor! ** [ 0.158012] ** ** [ 0.159010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160009] ************************************************************* [ 0.161788] NET: Registered protocol family 16 [ 0.162417] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.164050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.165471] cpuidle: using governor menu [ 0.168111] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170476] PCI: Using configuration type 1 for base access [ 0.173123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.181085] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.182044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.183166] cryptd: max_cpu_qlen set to 1000 [ 0.184244] ACPI: Added _OSI(Module Device) [ 0.185011] ACPI: Added _OSI(Processor Device) [ 0.186022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.187015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.191078] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.193575] ACPI: Interpreter enabled [ 0.194051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.195008] ACPI: Using IOAPIC for interrupt routing [ 0.196111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.197363] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.205427] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.206060] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.207019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.208084] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.210276] acpiphp: Slot [2] registered [ 0.211107] acpiphp: Slot [5] registered [ 0.212104] acpiphp: Slot [6] registered [ 0.213096] acpiphp: Slot [3] registered [ 0.214097] acpiphp: Slot [4] registered [ 0.215081] acpiphp: Slot [7] registered [ 0.216334] acpiphp: Slot [8] registered [ 0.217089] acpiphp: Slot [9] registered [ 0.218104] acpiphp: Slot [10] registered [ 0.219084] acpiphp: Slot [11] registered [ 0.220090] acpiphp: Slot [12] registered [ 0.221093] acpiphp: Slot [13] registered [ 0.222079] acpiphp: Slot [14] registered [ 0.223082] acpiphp: Slot [15] registered [ 0.224108] acpiphp: Slot [16] registered [ 0.225104] acpiphp: Slot [17] registered [ 0.226168] acpiphp: Slot [18] registered [ 0.228107] acpiphp: Slot [19] registered [ 0.229111] acpiphp: Slot [20] registered [ 0.231095] acpiphp: Slot [21] registered [ 0.232094] acpiphp: Slot [22] registered [ 0.233090] acpiphp: Slot [23] registered [ 0.235125] acpiphp: Slot [24] registered [ 0.236114] acpiphp: Slot [25] registered [ 0.238124] acpiphp: Slot [26] registered [ 0.240094] acpiphp: Slot [27] registered [ 0.241132] acpiphp: Slot [28] registered [ 0.243122] acpiphp: Slot [29] registered [ 0.245132] acpiphp: Slot [30] registered [ 0.247095] acpiphp: Slot [31] registered [ 0.248096] PCI host bridge to bus 0000:00 [ 0.249015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.252024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.254022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.257029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.259025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.262035] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264214] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.268169] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.273522] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.281019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.286054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.290021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.293019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.296018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.298676] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.301807] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.305082] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.308813] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.312015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.323788] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.329017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.337721] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.346017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.353035] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.372014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.384795] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390921] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.397015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.418017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.428978] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.433442] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.436383] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.440453] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.445266] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.448475] iommu: Default domain type: Passthrough [ 0.450466] SCSI subsystem initialized [ 0.452124] ACPI: bus type USB registered [ 0.453107] usbcore: registered new interface driver usbfs [ 0.455071] usbcore: registered new interface driver hub [ 0.457080] usbcore: registered new device driver usb [ 0.459191] pps_core: LinuxPPS API ver. 1 registered [ 0.460008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.463066] PTP clock support registered [ 0.465096] EDAC MC: Ver: 3.0.0 [ 0.467211] PCI: Using ACPI for IRQ routing [ 0.469758] NetLabel: Initializing [ 0.471018] NetLabel: domain hash size = 128 [ 0.474013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.475081] NetLabel: unlabeled traffic allowed by default [ 0.478168] vgaarb: loaded [ 0.479314] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.481015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.488000] clocksource: Switched to clocksource kvm-clock [ 0.591351] VFS: Disk quotas dquot_6.6.0 [ 0.592950] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.595671] *** VALIDATE ramfs *** [ 0.597016] *** VALIDATE hugetlbfs *** [ 0.598845] pnp: PnP ACPI init [ 0.601240] pnp: PnP ACPI: found 6 devices [ 0.619641] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.623237] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.625673] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.628489] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.631291] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.633802] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.637283] NET: Registered protocol family 2 [ 0.640236] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.647353] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.652080] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.657930] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.662872] TCP: Hash tables configured (established 65536 bind 65536) [ 0.672077] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.677962] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.685648] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.690639] NET: Registered protocol family 1 [ 0.693526] RPC: Registered named UNIX socket transport module. [ 0.696049] RPC: Registered udp transport module. [ 0.699683] RPC: Registered tcp transport module. [ 0.702960] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.706943] NET: Registered protocol family 44 [ 0.708229] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.710016] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.712654] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.715730] PCI: CLS 0 bytes, default 64 [ 0.717548] Unpacking initramfs... [ 3.375553] debug: unmapping init [mem 0xffff98a03cc64000-0xffff98a03ffcffff] [ 3.381441] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.386067] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.391949] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.308941] Initialise system trusted keyrings [ 4.313454] Key type blacklist registered [ 4.321426] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.347649] zbud: loaded [ 4.359201] *** VALIDATE nfs *** [ 4.361588] *** VALIDATE nfs4 *** [ 4.365455] pstore: using deflate compression [ 4.372205] Platform Keyring initialized [ 4.635477] NET: Registered protocol family 38 [ 4.638816] Key type asymmetric registered [ 4.642515] Asymmetric key parser 'x509' registered [ 4.644909] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.649541] io scheduler mq-deadline registered [ 4.656102] io scheduler kyber registered [ 4.660365] io scheduler bfq registered [ 4.665202] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.672232] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.685172] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.688497] ACPI: Power Button [PWRF] [ 4.698817] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.726583] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.754240] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.787805] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.820372] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.826687] Non-volatile memory driver v1.3 [ 4.828561] Linux agpgart interface v0.103 [ 4.872548] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 4.879579] vda: detected capacity change from 0 to 74829824 [ 4.901452] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.904408] vdb: detected capacity change from 0 to 1073741824 [ 4.923051] libphy: Fixed MDIO Bus: probed [ 4.928423] usbcore: registered new interface driver usbserial_generic [ 4.932353] usbserial: USB Serial support registered for generic [ 4.936164] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.943493] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.947196] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.953151] mousedev: PS/2 mouse device common for all mice [ 4.975417] rtc_cmos 00:05: RTC can wake from S4 [ 4.985276] rtc_cmos 00:05: registered as rtc0 [ 4.987675] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.990238] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.999601] intel_pstate: CPU model not supported [ 5.012400] hid: raw HID events driver (C) Jiri Kosina [ 5.014772] usbcore: registered new interface driver usbhid [ 5.022211] usbhid: USB HID core driver [ 5.028153] drop_monitor: Initializing network drop monitor service [ 5.028607] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.037661] Initializing XFRM netlink socket [ 5.038213] NET: Registered protocol family 10 [ 5.044854] Segment Routing with IPv6 [ 5.055087] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.063210] NET: Registered protocol family 17 [ 5.082841] mpls_gso: MPLS GSO support [ 5.094534] RAS: Correctable Errors collector initialized. [ 5.101460] AVX version of gcm_enc/dec engaged. [ 5.103614] AES CTR mode by8 optimization enabled [ 5.238414] sched_clock: Marking stable (5238177278, 0)->(6152476019, -914298741) [ 5.251763] registered taskstats version 1 [ 5.255194] Loading compiled-in X.509 certificates [ 5.260872] zswap: loaded using pool lzo/zbud [ 5.337596] Key type big_key registered [ 5.376119] Key type encrypted registered [ 5.377809] ima: No TPM chip found, activating TPM-bypass! [ 5.381842] ima: Allocated hash algorithm: sha1 [ 5.384648] ima: No architecture policies found [ 5.387681] evm: Initialising EVM extended attributes: [ 5.438082] evm: security.selinux [ 5.446995] evm: security.ima [ 5.451777] evm: security.capability [ 5.457027] evm: HMAC attrs: 0x1 [ 5.484134] rtc_cmos 00:05: setting system clock to 2026-08-24 20:34:32 UTC (1787603672) [ 5.497194] debug: unmapping init [mem 0xffffffffa9803000-0xffffffffa99fffff] [ 5.506339] debug: unmapping init [mem 0xffffffffa8582000-0xffffffffa8858fff] [ 5.528218] Write protecting the kernel read-only data: 28672k [ 5.534795] debug: unmapping init [mem 0xffffffffa6c03000-0xffffffffa6dfffff] [ 5.539697] debug: unmapping init [mem 0xffffffffa7514000-0xffffffffa75fffff] [ 5.635304] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.656820] systemd[1]: Detected virtualization kvm. [ 5.659642] systemd[1]: Detected architecture x86-64. [ 5.662968] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.719536] systemd[1]: No hostname configured. [ 5.723303] systemd[1]: Set hostname to . [ 5.726860] random: systemd: uninitialized urandom read (16 bytes read) [ 5.730692] systemd[1]: Initializing machine ID from random generator. [ 5.844827] random: ln: uninitialized urandom read (6 bytes read) [ 6.057951] random: systemd: uninitialized urandom read (16 bytes read) [ 6.062582] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 6.070474] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 6.081411] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Setup Virtual Console... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.360816] device-mapper: uevent: version 1.0.3 [ 7.370497] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook...[ 8.752777] random: fast init done [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.906313] virtio_net virtio0 ens2: renamed from eth0 [ 9.141676] scsi host0: ata_piix [ 9.193781] scsi host1: ata_piix [ 9.198846] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 9.215414] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 13.988613] random: crng init done [ 13.992086] random: 7 urandom warning(s) missed due to ratelimiting [ 15.338272] dracut-initqueue[595]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.824292] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.378362] printk: systemd: 25 output lines suppressed due to ratelimiting [ 20.214625] SELinux: Disabled at runtime. [ 20.327052] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 20.347605] systemd[1]: Detected virtualization kvm. [ 20.352120] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 22.241584] systemd[1]: initrd-switch-root.service: Succeeded. [ 22.247730] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 22.267817] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 22.280303] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 22.295333] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 22.325791] systemd[1]: Starting Journal Service... Starting Journal Service... [ 22.346748] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ 23.067566] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.065461] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.194771] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.255574] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 25.576040] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.630629] EDAC sbridge: Ver: 1.1.2 [ 28.910961] Key type dns_resolver registered [ 29.557765] NFS: Registering the id_resolver key type [ 29.562679] Key type id_resolver registered [ 29.565150] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. [ 33.698009] hrtimer: interrupt took 14475323 ns Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg433-client login: [ 98.188425] libcfs: loading out-of-tree module taints kernel. [ 98.383114] Key type ._llcrypt registered [ 98.392314] Key type .llcrypt registered [ 98.947836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 98.975231] alg: No test for adler32 (adler32-zlib) [ 100.653959] Lustre: Lustre: Build Version: 2.17.57_80_ged04d84 [ 101.805814] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 103.576941] Key type lgssc registered [ 105.466779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 234.183780] Lustre: Mounted lustre-client - version 2.17.57_80_ged04d84 [ 238.468328] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 251.712650] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing check_logdir /tmp/testlogs/ [ 256.827880] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing yml_node [ 260.071711] Lustre: lustre-OST0000-osc-ffff98a082c04000: disconnect after 24s idle [ 260.988905] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 263.765362] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 266.020870] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 267.589928] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 16:38:53 EDT 2026 [ 281.749632] Lustre: Unmounted lustre-client [ 312.403987] Key type lgssc unregistered [ 312.742380] LNet: 8421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.760853] LNetError: 8421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.794223] LNet: Removed LNI 192.168.204.33@tcp [ 313.704906] Key type .llcrypt unregistered [ 313.708829] Key type ._llcrypt unregistered [ 341.546713] Key type ._llcrypt registered [ 341.549648] Key type .llcrypt registered [ 341.766532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.775979] alg: No test for adler32 (adler32-zlib) [ 342.672924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 342.744475] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 342.757936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 347.052662] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 347.255530] LNet: 9305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.262101] LNetError: 9305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 347.295778] LNet: Removed LNI 192.168.204.33@tcp [ 348.464657] Key type .llcrypt unregistered [ 348.475937] Key type ._llcrypt unregistered [ 355.709561] Key type ._llcrypt registered [ 355.711359] Key type .llcrypt registered [ 357.042445] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:40:22 (1787604022) [ 357.497358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 357.512719] alg: No test for adler32 (adler32-zlib) [ 358.400453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.492648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.517762] LNet: 9727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.539186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.595163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 358.616267] LNet: 9734:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 366.794508] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:40:32 (1787604032) [ 366.972889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.011279] LNet: 10310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.038889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.156627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 374.027716] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:40:39 (1787604039) [ 374.235928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.257405] LNet: 10898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.279769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.379096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 382.010980] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:40:47 (1787604047) [ 382.372446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.403901] LNet: 11482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.438637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.610411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 389.211934] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:40:54 (1787604054) [ 389.518639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.552251] LNet: 12068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.585612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.713894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 390.461317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 398.058976] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:41:03 (1787604063) [ 398.169464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.211854] LNet: 12651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.259762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.445391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 399.165604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 399.931715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 407.990801] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:41:13 (1787604073) [ 408.122926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.198951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.352177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 415.965293] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:41:21 (1787604081) [ 416.074084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.113173] LNet: 13834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.121168] LNet: 13834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 416.150220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.200749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 416.248438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 416.835334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 416.897104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 417.532874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 417.577718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 418.191446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 418.236709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 418.890854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 418.940914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.507122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 419.571659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.112921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 420.151439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 420.538818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 420.585396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 421.096860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 421.126966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 421.602565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 421.660906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 422.269235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 422.293070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 422.546827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 422.591910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 423.027822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 423.087853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 431.202920] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:41:36 (1787604096) [ 431.278719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.358170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.459322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 431.494769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 439.412749] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:41:44 (1787604104) [ 439.572616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.634195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.737647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 439.781477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 447.064842] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:41:52 (1787604112) [ 447.202933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.270963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.350961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 447.388686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 453.412831] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:41:59 (1787604119) [ 453.786508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.820779] LNet: 16283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.829362] LNet: 16283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 453.854994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.920443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 453.960476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 459.934290] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:42:05 (1787604125) [ 460.147927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.194326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.331671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 460.383386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 467.791084] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:42:13 (1787604133) [ 468.099671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.136296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.211487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 468.244403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 476.255342] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:42:21 (1787604141) [ 476.469359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.541776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.637898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.101378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.446270] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:42:30 (1787604150) [ 485.525994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.562500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.861297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.058587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 493.623955] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:42:39 (1787604159) [ 493.736213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.813557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.973727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 494.178634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.088876] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:42:45 (1787604165) [ 500.167899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.225556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.730169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.009117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 508.680601] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:42:53 (1787604173) [ 508.801957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.868227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.040115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.112595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.147771] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:43:01 (1787604181) [ 516.332384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.395320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.496838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 516.554599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.777864] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:43:08 (1787604188) [ 522.951036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.973912] LNet: 21725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 522.987946] LNet: 21725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 523.025477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.135597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 528.679060] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:43:14 (1787604194) [ 528.746034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.795650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.883748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 534.872919] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:43:20 (1787604200) [ 534.979904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 535.033932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.152819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 535.648144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 541.158530] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:43:26 (1787604206) [ 541.217281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.242244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.308325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 541.335515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 546.753454] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:43:32 (1787604212) [ 546.970771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.005535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.109387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 547.161666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 553.054320] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:43:39 (1787604219) [ 553.210903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.270564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.331973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 553.367727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 559.945948] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:43:45 (1787604225) [ 560.084401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.137109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.213404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 560.248039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 567.003735] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:43:52 (1787604232) [ 567.121590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.183375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.284607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 567.422834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 567.460961] LNetError: 25840:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 567.516793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 567.576642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 574.481501] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:43:59 (1787604239) [ 574.644436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.691456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.780283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lEFtICJJSZ [ 574.861242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lEFtICJJSZ [ 574.928201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.lEFtICJJSZ [ 581.146637] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:44:06 (1787604246) [ 581.356504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.429585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.520996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 581.576505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 581.649785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 589.237118] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 16:44:14 (1787604254) [ 589.368105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.406193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.484395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.506776] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 591.494464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 592.883908] Lustre: Lustre: Build Version: 2.17.57_80_ged04d84 [ 594.767242] Key type lgssc registered [ 596.287424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 601.091719] LNetError: 27601:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.33@tcp99 has no available nets [ 604.255473] Key type lgssc unregistered [ 604.572339] LNetError: 28712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.599698] LNet: Removed LNI 192.168.204.33@tcp [ 605.269659] Key type .llcrypt unregistered [ 605.273445] Key type ._llcrypt unregistered [ 612.706786] Key type ._llcrypt registered [ 612.718123] Key type .llcrypt registered [ 614.187480] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:44:39 (1787604279) [ 614.908315] Key type .llcrypt unregistered [ 614.910830] Key type ._llcrypt unregistered [ 621.516836] Key type ._llcrypt registered [ 621.519348] Key type .llcrypt registered [ 621.745518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 621.767318] alg: No test for adler32 (adler32-zlib) [ 622.839943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.003205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 623.021640] LNet: 29633:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 623.032454] LNet: 29633:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 623.052945] LNet: 29633:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 623.068690] LNet: Message forwarding will be enabled in 2s [ 626.143229] LNet: Message forwarding enabled [ 626.890919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 626.932667] LNet: Message forwarding will stop in 2s to 6s [ 630.252426] LNet: Message forwarding disabled [ 630.680981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 630.734319] LNet: 29722:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 630.741976] LNet: 29722:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 630.749638] LNet: 29722:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 630.762926] LNet: Message forwarding will be enabled in 2s [ 631.027240] LNet: 29735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.068631] Key type .llcrypt unregistered [ 632.075124] Key type ._llcrypt unregistered [ 648.066070] Key type ._llcrypt registered [ 648.073643] Key type .llcrypt registered [ 649.583717] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:45:14 (1787604314) [ 649.944730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 649.953044] alg: No test for adler32 (adler32-zlib) [ 650.854409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.971293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 658.815923] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:45:24 (1787604324) [ 659.176685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.205505] LNet: 31142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.225533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.310765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 659.341907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 659.363867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 659.409402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 659.471591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 659.532580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 659.558317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 659.582996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 659.608361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 659.647110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 659.700879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 659.752458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 659.787770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 659.855986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 659.916556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 659.992915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 660.046270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 660.077774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 660.104984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 660.120977] LNetError: 31185:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 667.310972] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:45:32 (1787604332) [ 667.429840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.448849] LNet: 31750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.483233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.598178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.785073] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:45:41 (1787604341) [ 677.781535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg433-server [ 679.879382] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:45:45 (1787604345) [ 680.217726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.255831] LNet: 32689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.298708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.395804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 680.448802] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 681.322393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[34-38]@tcp [ 681.342674] LNet: 32822:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 681.378088] LNetError: 32693:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.34@tcp is being used as a gateway but routing feature is not turned on [ 681.398147] LNetError: 32693:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.34@tcp has gone from up to down [ 688.708229] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:45:54 (1787604354) [ 690.423714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg433-server [ 692.209412] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:45:57 (1787604357) [ 692.329528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.353752] LNet: 33759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.362662] LNetError: 33759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.391874] LNet: Removed LNI 192.168.204.33@tcp [ 692.431492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.473863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 692.493135] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 693.095554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[34-39/2]@tcp [ 693.112394] LNet: 33877:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 693.121268] LNet: 33877:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 693.146573] LNetError: 33764:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.34@tcp is being used as a gateway but routing feature is not turned on [ 693.158197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[34-39/2]@tcp [ 693.161197] LNetError: 33764:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 693.161224] LNetError: 33764:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.34@tcp has gone from up to down [ 693.161229] LNetError: 33764:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 700.646932] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:46:05 (1787604365) [ 700.781663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.808440] LNet: 34464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.826243] LNetError: 34464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.851322] LNet: Removed LNI 192.168.204.33@tcp [ 700.922211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.052205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 701.132969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.192599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.303506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.387033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.430795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.490402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 701.623522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.670526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.809403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 701.923525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.976684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.064860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 702.198840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.250391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.362115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.547771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.620427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.680761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 702.771581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.818437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.889920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 702.959390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.993476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.081363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 703.178936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.240167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.356972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.456190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.502221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.599431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 703.660542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.695654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.760377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.244567] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:46:15 (1787604375) [ 711.971122] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg433-server [ 714.279363] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:46:19 (1787604379) [ 716.317710] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg433-server [ 717.759868] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:46:23 (1787604383) [ 718.010398] LNet: 35871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.016973] LNet: 35871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 718.594747] Key type .llcrypt unregistered [ 718.601735] Key type ._llcrypt unregistered [ 724.961483] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 725.086343] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 725.097996] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.102370] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.799532] Key type ._llcrypt registered [ 727.801376] Key type .llcrypt registered [ 728.023049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 728.038919] alg: No test for adler32 (adler32-zlib) [ 728.997854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.080761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 729.098096] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 729.759441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.780630] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 731.720549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 731.759932] LNet: Removed LNI 10.1.2.3@tcp [ 731.899836] LNet: 36594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 731.905327] LNetError: 36594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 731.927192] LNet: Removed LNI 192.168.204.33@tcp [ 732.437606] Key type .llcrypt unregistered [ 732.439744] Key type ._llcrypt unregistered [ 746.535199] Key type ._llcrypt registered [ 746.539784] Key type .llcrypt registered [ 748.149845] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:46:53 (1787604413) [ 748.715324] Key type .llcrypt unregistered [ 748.717342] Key type ._llcrypt unregistered [ 756.405744] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.615568] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.621294] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.648413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 759.503930] Key type ._llcrypt registered [ 759.509121] Key type .llcrypt registered [ 759.656936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.672322] alg: No test for adler32 (adler32-zlib) [ 760.701491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.835495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 760.893572] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 762.258396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.288173] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.213053] LNet: 38098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.226690] LNetError: 38098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 765.232460] LNet: Removed LNI 10.1.2.3@tcp [ 765.241737] LNet: Removed LNI 192.168.204.33@tcp [ 766.082726] Key type .llcrypt unregistered [ 766.084192] Key type ._llcrypt unregistered [ 778.444391] Key type ._llcrypt registered [ 778.451183] Key type .llcrypt registered [ 779.744446] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:47:25 (1787604445) [ 780.514523] Key type .llcrypt unregistered [ 780.522516] Key type ._llcrypt unregistered [ 787.563902] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 787.734167] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 787.740377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 787.746241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 790.399585] Key type ._llcrypt registered [ 790.402027] Key type .llcrypt registered [ 790.650450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 790.684511] alg: No test for adler32 (adler32-zlib) [ 791.593956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.654902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 791.697498] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 793.792603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 793.811577] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 796.268944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 796.293346] LNet: Removed LNI 10.1.2.3@tcp [ 796.316982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 796.354314] LNet: Removed LNI 10.1.2.31@tcp [ 796.371670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.401289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 797.177985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 797.199677] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 798.367944] LNet: 40027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.373744] LNetError: 40027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 798.381900] LNet: Removed LNI 10.1.2.31@tcp [ 798.393315] LNet: Removed LNI 10.1.2.3@tcp [ 798.995628] Key type .llcrypt unregistered [ 798.998596] Key type ._llcrypt unregistered [ 812.219059] Key type ._llcrypt registered [ 812.220721] Key type .llcrypt registered [ 814.102095] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:47:58 (1787604478) [ 814.516218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 814.539817] alg: No test for adler32 (adler32-zlib) [ 815.467679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.530976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.563348] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 816.321226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.352342] LNet: 40984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.370540] LNetError: 40984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.382506] LNet: Removed LNI 192.168.204.33@tcp [ 816.401345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.456255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.33@tcp [ 816.485463] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 817.254519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.33@tcp [ 817.291243] LNet: Removed LNI 192.168.204.33@tcp [ 817.369551] LNet: 41119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.992501] Key type .llcrypt unregistered [ 817.997237] Key type ._llcrypt unregistered [ 830.393400] Key type ._llcrypt registered [ 830.396753] Key type .llcrypt registered [ 831.903561] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:48:17 (1787604497) [ 834.153988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg433-server [ 836.302288] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:48:21 (1787604501) [ 837.002789] Key type .llcrypt unregistered [ 837.004726] Key type ._llcrypt unregistered [ 844.957879] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 845.116973] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 845.124397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 845.130565] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 848.126507] Key type ._llcrypt registered [ 848.131170] Key type .llcrypt registered [ 848.395493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 848.420752] alg: No test for adler32 (adler32-zlib) [ 849.395432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.486865] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 850.178786] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 852.925938] LNet: 42979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.932961] LNetError: 42979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.938845] LNet: Removed LNI 192.168.204.33@tcp1 [ 852.945526] LNet: Removed LNI 10.1.2.3@tcp2 [ 853.602546] Key type .llcrypt unregistered [ 853.605281] Key type ._llcrypt unregistered [ 866.385896] Key type ._llcrypt registered [ 866.391589] Key type .llcrypt registered [ 867.946267] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:48:53 (1787604533) [ 868.355445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 868.376749] alg: No test for adler32 (adler32-zlib) [ 869.291661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.346958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.375381] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 870.140908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.158862] LNet: 43953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.164707] LNetError: 43953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.175786] LNet: Removed LNI 192.168.204.33@tcp [ 870.206886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.307461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 870.333308] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 877.559185] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:49:03 (1787604543) [ 877.812288] LNet: 44544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.818894] LNetError: 44544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.841665] LNet: Removed LNI 192.168.204.33@tcp [ 878.417668] Key type .llcrypt unregistered [ 878.419420] Key type ._llcrypt unregistered [ 884.789250] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 884.911057] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 884.914570] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 884.917939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 887.960638] Key type ._llcrypt registered [ 887.962787] Key type .llcrypt registered [ 888.276401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 888.310209] alg: No test for adler32 (adler32-zlib) [ 889.366708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.452542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 889.504753] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 890.671772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 890.714933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.416918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.455692] LNet: 45289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.460496] LNetError: 45289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 893.470536] LNet: Removed LNI 10.1.2.3@tcp [ 893.479842] LNet: Removed LNI 192.168.204.33@tcp [ 893.513512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.611334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 893.644791] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 893.653403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.251551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.272593] LNet: 45316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.280234] LNetError: 45316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 894.285928] LNetError: 45316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 894.292901] LNet: Removed LNI 10.1.2.3@tcp [ 894.306186] LNet: Removed LNI 192.168.204.33@tcp [ 894.339969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.430742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 894.449838] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 894.456884] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 894.953419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 894.984765] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 896.096716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.124992] LNet: 45479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.134811] LNetError: 45479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 896.145148] LNetError: 45479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 896.157548] LNet: Removed LNI 10.1.2.3@tcp [ 896.163925] LNet: Removed LNI 192.168.204.33@tcp [ 896.176036] LNet: Removed LNI 192.168.204.33@tcp2 [ 896.195926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.275207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 896.307677] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 896.312982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 896.320677] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 905.002750] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:49:30 (1787604570) [ 905.385453] LNet: 46078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.390554] LNetError: 46078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 905.398791] LNetError: 46078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 905.408227] LNet: Removed LNI 10.1.2.3@tcp [ 905.411662] LNet: Removed LNI 192.168.204.33@tcp [ 905.425929] LNet: Removed LNI 192.168.204.33@tcp2 [ 906.265743] Key type .llcrypt unregistered [ 906.269808] Key type ._llcrypt unregistered [ 913.172073] Key type ._llcrypt registered [ 913.176466] Key type .llcrypt registered [ 913.452654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.488902] alg: No test for adler32 (adler32-zlib) [ 914.492617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.571758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 914.604531] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 915.672438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.726723] LNet: 46477:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.739216] LNetError: 46477:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.764795] LNet: Removed LNI 192.168.204.33@tcp [ 915.800567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.934409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 915.973225] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 916.759329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.790329] LNet: 46507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.798285] LNetError: 46507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.809767] LNet: Removed LNI 192.168.204.33@tcp [ 916.830934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.872460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 916.889698] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 917.358219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.386531] LNet: Removed LNI 192.168.204.33@tcp [ 917.417087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.495103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 917.538623] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 917.980288] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 918.111365] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 918.120165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 918.132500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 920.587665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.608126] LNet: 46616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.611929] LNet: 46616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 920.617887] LNetError: 46616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.621637] LNetError: 46616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 920.634315] LNet: Removed LNI 192.168.204.33@tcp [ 920.676897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.719860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 920.738528] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 928.998883] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:49:54 (1787604594) [ 929.350559] LNet: 47213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.363605] LNetError: 47213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.386801] LNet: Removed LNI 192.168.204.33@tcp [ 930.760311] Key type .llcrypt unregistered [ 930.768389] Key type ._llcrypt unregistered [ 938.858753] Key type ._llcrypt registered [ 938.867093] Key type .llcrypt registered [ 939.184267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.228036] alg: No test for adler32 (adler32-zlib) [ 940.252519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 940.398041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 940.431848] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 941.602296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.634645] LNet: 47616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.640388] LNetError: 47616:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.666569] LNet: Removed LNI 192.168.204.33@tcp [ 941.726609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.850172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 941.874779] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 949.528443] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:50:14 (1787604614) [ 949.635093] LNet: 48200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.695439] LNetError: 48200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.716040] LNet: Removed LNI 192.168.204.33@tcp [ 950.361588] Key type .llcrypt unregistered [ 950.363586] Key type ._llcrypt unregistered [ 956.770044] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 956.874422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 956.888471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 956.897253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 959.472425] Key type ._llcrypt registered [ 959.477326] Key type .llcrypt registered [ 959.751749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.766820] alg: No test for adler32 (adler32-zlib) [ 960.629787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.670438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.723720] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 961.666804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.697594] LNet: 48662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.704419] LNetError: 48662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.770777] LNet: Removed LNI 192.168.204.33@tcp [ 961.791328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.870465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 961.907732] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 969.886572] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:50:34 (1787604634) [ 970.031560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.061950] LNet: 49253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.073166] LNetError: 49253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.091727] LNet: Removed LNI 192.168.204.33@tcp [ 970.143173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.320749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 977.829846] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:50:42 (1787604642) [ 977.987387] LNet: 49835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.921530] Key type .llcrypt unregistered [ 978.928780] Key type ._llcrypt unregistered [ 985.316354] Key type ._llcrypt registered [ 985.323774] Key type .llcrypt registered [ 985.607336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 985.633205] alg: No test for adler32 (adler32-zlib) [ 986.624888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.732402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 986.769470] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 987.732576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.744951] LNet: 50243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 987.763837] LNetError: 50243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 987.790851] LNet: Removed LNI 192.168.204.33@tcp [ 987.849882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.951167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 987.996812] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 999.542478] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:51:03 (1787604663) [ 999.764063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 999.835556] LNet: 50832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.854661] LNetError: 50832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.888946] LNet: Removed LNI 192.168.204.33@tcp [ 999.981332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.120660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1011.437378] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:51:16 (1787604676) [ 1013.893729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1015.847270] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:51:21 (1787604681) [ 1018.196649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1020.968801] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:51:26 (1787604686) [ 1022.769757] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1024.923455] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:51:30 (1787604690) [ 1026.633346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1029.261923] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:51:34 (1787604694) [ 1031.409136] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1032.978287] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 16:51:38 (1787604698) [ 1033.366437] LNet: 52892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.000599] Key type .llcrypt unregistered [ 1034.002539] Key type ._llcrypt unregistered [ 1040.604506] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1040.858247] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1040.882763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1040.892123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1043.983839] Key type ._llcrypt registered [ 1043.989305] Key type .llcrypt registered [ 1044.283613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.325465] alg: No test for adler32 (adler32-zlib) [ 1045.446737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.626633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1045.678821] LNet: Added LNI 192.168.204.33@tcp [8/512/0/180] [ 1045.697993] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1045.700878] LNet: 53212:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1045.915437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1045.957242] LNet: 53231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.966907] LNetError: 53231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1045.979588] LNet: Removed LNI 10.1.2.3@tcp [ 1045.985957] LNetError: 53231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.015937] LNet: Removed LNI 192.168.204.33@tcp [ 1046.066798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.185883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.215819] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1046.321054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.347086] LNet: 53256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.353070] LNetError: 53256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.374698] LNet: Removed LNI 192.168.204.33@tcp [ 1046.395552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.455294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.477358] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1046.486313] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1046.488394] LNet: 53264:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1046.613852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.634774] LNet: 53282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.641147] LNetError: 53282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1046.647791] LNet: Removed LNI 10.1.2.3@tcp [ 1046.650095] LNetError: 53282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.663956] LNet: Removed LNI 192.168.204.33@tcp [ 1046.703372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.791202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1046.835853] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1046.851415] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1046.857027] LNet: 53290:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.120063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.147108] LNet: 53316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.155658] LNetError: 53316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.161215] LNet: Removed LNI 10.1.2.3@tcp [ 1047.164022] LNetError: 53316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.184670] LNet: Removed LNI 192.168.204.33@tcp [ 1047.214727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.307824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.349248] LNet: Added LNI 192.168.204.33@tcp [8/256/0/182] [ 1047.353516] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1047.459585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1047.483633] LNet: 53344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.489751] LNetError: 53344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.495540] LNet: Removed LNI 10.1.2.3@tcp [ 1047.497768] LNetError: 53344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.509911] LNet: Removed LNI 192.168.204.33@tcp [ 1047.532935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1047.626882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1047.653305] LNet: Added LNI 192.168.204.33@tcp [8/256/0/182] [ 1047.658710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1047.660809] LNet: 53352:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1047.749376] LNet: Removed LNI 10.1.2.3@tcp [ 1047.761920] LNet: Removed LNI 192.168.204.33@tcp [ 1048.716570] Key type .llcrypt unregistered [ 1048.720918] Key type ._llcrypt unregistered [ 1063.885681] Key type ._llcrypt registered [ 1063.888743] Key type .llcrypt registered [ 1065.266384] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 16:52:10 (1787604730) [ 1067.085306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1069.142889] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:52:14 (1787604734) [ 1069.672084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.690140] alg: No test for adler32 (adler32-zlib) [ 1070.607705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1070.642064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1070.718766] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1071.929796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.33 [ 1071.966823] LNet: 54620:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1071.991145] LNet: Removed LNI 192.168.204.33@tcp [ 1079.278561] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:52:24 (1787604744) [ 1079.369211] LNet: 55183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.379854] LNetError: 55183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.404527] LNet: Removed LNI 192.168.204.33@tcp [ 1080.100616] Key type .llcrypt unregistered [ 1080.102302] Key type ._llcrypt unregistered [ 1085.978639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1086.224220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1088.109708] Key type ._llcrypt registered [ 1088.112784] Key type .llcrypt registered [ 1088.439724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1088.471640] alg: No test for adler32 (adler32-zlib) [ 1089.983310] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1096.781460] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:52:42 (1787604762) [ 1096.956951] LNet: 56099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1096.961995] LNetError: 56099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1096.978238] LNet: Removed LNI 10.1.2.3@tcp [ 1097.647147] Key type .llcrypt unregistered [ 1097.651168] Key type ._llcrypt unregistered [ 1105.441093] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1105.733431] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1105.744628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.759878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1108.770990] Key type ._llcrypt registered [ 1108.773573] Key type .llcrypt registered [ 1109.001049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1109.026565] alg: No test for adler32 (adler32-zlib) [ 1110.080088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1118.773882] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:53:03 (1787604783) [ 1120.688357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1122.695680] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:53:08 (1787604788) [ 1122.867345] LNet: 57281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.882825] LNetError: 57281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1122.905382] LNet: Removed LNI 10.1.2.3@tcp [ 1123.961804] Key type .llcrypt unregistered [ 1123.964578] Key type ._llcrypt unregistered [ 1130.878537] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1131.148700] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1131.154318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1131.158872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1134.189911] Key type ._llcrypt registered [ 1134.194483] Key type .llcrypt registered [ 1134.476136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1134.495850] alg: No test for adler32 (adler32-zlib) [ 1135.484241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1135.539726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1145.138805] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:53:30 (1787604810) [ 1145.771764] LNet: 58524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.784638] LNetError: 58524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1145.815642] LNet: Removed LNI 10.1.2.3@tcp [ 1146.845442] Key type .llcrypt unregistered [ 1146.849122] Key type ._llcrypt unregistered [ 1153.212060] Key type ._llcrypt registered [ 1153.219784] Key type .llcrypt registered [ 1153.566894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1153.600793] alg: No test for adler32 (adler32-zlib) [ 1154.593801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1154.621205] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1163.480936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1164.391445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1164.469969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1164.527280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1164.556843] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1164.824221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.870832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1164.897334] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1164.912135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.950819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.006456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.062724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.135496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.193201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.215832] LNet: 58966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.241457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.288847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.319368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.364322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.434594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.500958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.551885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1166.264407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1166.308557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1166.340166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1166.371755] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1166.627559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.661595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.723923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.783497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.865861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.932530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.979432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.030719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.057994] LNet: 59076:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1167.087741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.116185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.142826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.178754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.221580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.281216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.330256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.888642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1167.918856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1167.951572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1167.968433] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1168.181908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.208312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.231708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.274573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.297372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.322345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.355850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.420074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.449548] LNet: 59185:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.478069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.532818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.589121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.648659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.701302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.738535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.798638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.462195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1169.525574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1169.585182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1169.857653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.927228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.990753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.028860] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1170.039593] LNet: Skipped 1 previous similar message [ 1170.041616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.102139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.147281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.211920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.256091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.301785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.360273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.430705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.478398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.513112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.544870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.574575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.139600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1171.212645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1171.273166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1171.535581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.567775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.610190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.653418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.708442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.771528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.837970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.876563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.891988] LNet: 59404:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.896893] LNet: 59404:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1171.911292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.934303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.957692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.986836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.013665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.036901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.068935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.517761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1172.551955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1172.581975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1172.754424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.783958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.818507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1172.854642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1172.891960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1172.926682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.970526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.001467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.022588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.052686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.075939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.102650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.127487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.158719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.193920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1173.582738] LNet: 59558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.592743] LNetError: 59558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.615935] LNet: Removed LNI 192.168.204.33@tcp [ 1174.098424] Key type .llcrypt unregistered [ 1174.101729] Key type ._llcrypt unregistered [ 1192.212228] Key type ._llcrypt registered [ 1192.214319] Key type .llcrypt registered [ 1194.120761] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:54:19 (1787604859) [ 1194.942587] Key type .llcrypt unregistered [ 1194.945861] Key type ._llcrypt unregistered [ 1202.160222] Key type ._llcrypt registered [ 1202.164511] Key type .llcrypt registered [ 1202.523712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.543651] alg: No test for adler32 (adler32-zlib) [ 1203.518400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1203.585897] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1212.121204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1225.902251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.969448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1225.999509] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1227.760355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1227.818483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 1 [ 1227.882362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1227.941368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 1 [ 1228.015484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1228.069227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 1 [ 1228.115484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1228.166243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 1 [ 1228.196650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1228.217965] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1228.227290] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1228.537766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.564411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.597959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.655399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.714922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.766525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.809230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.858370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.881991] LNet: 61036:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.917330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.957237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1229.007775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1229.061976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1229.091672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1229.141822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1229.178564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1229.808233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1229.864550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 1 [ 1229.921500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1229.980139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 1 [ 1230.026777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1230.078872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 1 [ 1230.130867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1230.176915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 1 [ 1230.216605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1230.249639] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1230.254617] LNet: Skipped 2 previous similar messages [ 1230.594203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.639769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.664522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.701563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.737640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.768260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.800974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.833857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.862656] LNet: 61157:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.883269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.936728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.985466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.024560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.065504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.124612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.176726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1231.946087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1232.016573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 1 [ 1232.077977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1232.132695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 1 [ 1232.182246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1232.240056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 1 [ 1232.288849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1232.341388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 1 [ 1232.408876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1232.435788] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1232.443118] LNet: Skipped 2 previous similar messages [ 1232.810867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.866833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.920863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.965271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1233.009935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1233.061237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.096435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.145943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.170765] LNet: 61279:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.189857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.223774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.252024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.315369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.387927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.445519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.499305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1234.325888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1234.400651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 1 [ 1234.464949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1234.528786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 1 [ 1234.582627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1234.622833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 1 [ 1234.686125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1234.750834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 1 [ 1234.826442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1234.863459] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1234.875712] LNet: Skipped 2 previous similar messages [ 1235.110137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.144267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.185828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.222234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.248277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.275182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.301284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.327939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.345780] LNet: 61400:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.363822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.399157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.434000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.454983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.488152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.523199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.580820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.261243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1236.308900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 1 [ 1236.345913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1236.401927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 1 [ 1236.464917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1236.539457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 1 [ 1236.616449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1236.667688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 1 [ 1236.747212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1237.228441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.274462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.315299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.342280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.373713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.429941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.470153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.508836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.550394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.590562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.632894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.695926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.756370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.803641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.857366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.821036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1238.894878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_error -r 1 [ 1238.955779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1239.018682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_error -r 1 [ 1239.089184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1239.126878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_error -r 1 [ 1239.162314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1239.196664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_error -r 1 [ 1239.225234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1239.246754] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1239.251387] LNet: Skipped 5 previous similar messages [ 1239.480819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.538031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.602952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.658660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.724319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.757416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.797047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.847105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.875820] LNet: 61643:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.898105] LNet: 61643:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1239.923782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.989943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.030243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.082383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.116745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.183721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.285742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.540244] LNet: 61723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.545817] LNetError: 61723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.549049] LNet: Removed LNI 192.168.204.33@tcp [ 1246.559100] LNet: Removed LNI 192.168.204.33@tcp1 [ 1246.837710] Key type .llcrypt unregistered [ 1246.841409] Key type ._llcrypt unregistered [ 1269.050626] Key type ._llcrypt registered [ 1269.059531] Key type .llcrypt registered [ 1270.700535] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:55:35 (1787604935) [ 1271.855410] Key type .llcrypt unregistered [ 1271.858991] Key type ._llcrypt unregistered [ 1279.420471] Key type ._llcrypt registered [ 1279.423074] Key type .llcrypt registered [ 1279.675801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.706608] alg: No test for adler32 (adler32-zlib) [ 1280.651540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1280.693730] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1289.627484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1290.729811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1290.823379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1290.891556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1290.920248] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1291.328687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.405715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.448883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.485479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.524224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.555529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.583618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.645854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.683713] LNet: 62980:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.698597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.731510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.770609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.816366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.855631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.897955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.925239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1292.480918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1292.506852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1292.545113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1292.583248] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1292.931204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.967390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1293.005858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1293.037313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1293.096031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1293.168624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.241173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.305564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.325229] LNet: 63090:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.353681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.383699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.440709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.477542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.514322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.546333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.598949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1294.239363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1294.275901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1294.335432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1294.378609] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1294.689557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.747761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1294.780672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.833375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1294.894179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1294.951643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.004768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.051832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.080840] LNet: 63199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.107600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.176153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.247820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.271839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.307593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.333503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.356616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1295.607934] LNet: 63244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.613723] LNetError: 63244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.641240] LNet: Removed LNI 192.168.204.33@tcp [ 1296.145628] Key type .llcrypt unregistered [ 1296.150097] Key type ._llcrypt unregistered [ 1319.585919] Key type ._llcrypt registered [ 1319.587799] Key type .llcrypt registered [ 1321.334703] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:56:26 (1787604986) [ 1322.449440] Key type .llcrypt unregistered [ 1322.451180] Key type ._llcrypt unregistered [ 1328.506366] Key type ._llcrypt registered [ 1328.511985] Key type .llcrypt registered [ 1328.762923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1328.783886] alg: No test for adler32 (adler32-zlib) [ 1329.685920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1329.729335] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1338.160482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1348.091366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.169333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1348.201739] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1350.065744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1350.120178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 1 [ 1350.189125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1350.257672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 1 [ 1350.330512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1350.388486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 1 [ 1350.456455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1350.509749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 1 [ 1350.548280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1350.568162] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1350.575734] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1350.800948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.870528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1350.914198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.945354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.976145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1351.020835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1351.087389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1351.143915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1351.188548] LNet: 64718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1351.215069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1351.251443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.284167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.323488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.353928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.384439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.432278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1351.470823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1352.120976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1352.177172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_error -r 1 [ 1352.245991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1352.310825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_error -r 1 [ 1352.357022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1352.401189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_error -r 1 [ 1352.460129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1352.523796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_error -r 1 [ 1352.602500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1352.638582] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1352.645047] LNet: Skipped 2 previous similar messages [ 1352.956588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1352.988759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1353.021885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.065976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.123954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.164981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.207471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.258364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.289746] LNet: 64841:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.317914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1353.373985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1353.419944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.469990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.515126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.551504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.579255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1353.618055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.252541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1354.287707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_timeout -r 1 [ 1354.317462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1354.360372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_timeout -r 1 [ 1354.390881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1354.419382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_timeout -r 1 [ 1354.455883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1354.524171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_timeout -r 1 [ 1354.585067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1354.608881] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1354.910906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.959421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1354.993428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1355.022735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1355.059311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1355.098173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.141920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.177518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.199849] LNet: 64965:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1355.222930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.257037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.332490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.401415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.449133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.502701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.571187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1355.630727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.225942] LNet: 65047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.234423] LNetError: 65047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.240402] LNet: Removed LNI 192.168.204.33@tcp [ 1362.248320] LNet: Removed LNI 192.168.204.33@tcp1 [ 1362.583878] Key type .llcrypt unregistered [ 1362.587123] Key type ._llcrypt unregistered [ 1384.263374] Key type ._llcrypt registered [ 1384.265987] Key type .llcrypt registered [ 1385.613604] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:57:31 (1787605051) [ 1387.301213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1389.522942] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:57:34 (1787605054) [ 1390.321516] Key type .llcrypt unregistered [ 1390.325766] Key type ._llcrypt unregistered [ 1397.397439] Key type ._llcrypt registered [ 1397.402645] Key type .llcrypt registered [ 1397.735706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1397.756092] alg: No test for adler32 (adler32-zlib) [ 1398.762811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1398.801200] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1407.253673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1407.912581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1407.947928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1407.994965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1408.012848] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1408.032705] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1408.220714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1408.247521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1408.273794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1408.323822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1408.386336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1408.445607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1408.491321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1408.550229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1408.579368] LNet: 66598:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1408.625664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1408.666803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1408.716537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1408.781357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1408.826730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1408.864298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1409.091429] LNet: 66641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1409.097823] LNetError: 66641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1409.120214] LNet: Removed LNI 192.168.204.33@tcp [ 1409.562645] Key type .llcrypt unregistered [ 1409.567957] Key type ._llcrypt unregistered [ 1429.954580] Key type ._llcrypt registered [ 1429.964820] Key type .llcrypt registered [ 1430.258167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1430.276565] alg: No test for adler32 (adler32-zlib) [ 1431.203929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1431.237777] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1438.965495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1446.883241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.944417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1446.994284] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1448.325390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1448.366672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e network_timeout -r 1 [ 1448.413749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1448.456315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e network_timeout -r 1 [ 1448.507920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1448.550949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e network_timeout -r 1 [ 1448.605594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1448.636294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e network_timeout -r 1 [ 1448.688543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1448.712614] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1448.929495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1448.959902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1449.002747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1449.022908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1449.054094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1449.083205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1449.112305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1449.137658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1449.158513] LNet: 67526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1449.177915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1449.214283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1449.241827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1449.285682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1449.326390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1449.363305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1453.543836] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1453.791140] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1453.793630] LNet: 67604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.800396] LNetError: 67604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.819934] LNet: Removed LNI 192.168.204.33@tcp [ 1453.838273] LNet: Removed LNI 192.168.204.33@tcp1 [ 1454.393538] Key type .llcrypt unregistered [ 1454.395830] Key type ._llcrypt unregistered [ 1470.496256] Key type ._llcrypt registered [ 1470.498711] Key type .llcrypt registered [ 1472.431231] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:58:57 (1787605137) [ 1473.450431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1473.482468] alg: No test for adler32 (adler32-zlib) [ 1474.529792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.634780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.676442] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1475.647437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1475.687068] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1477.547382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1477.699780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1477.792518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1477.851916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1477.926564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1477.998190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1478.086254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1478.114253] LNetError: 68704:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1478.687525] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1478.687785] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1480.742763] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1482.338601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1484.832208] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1485.577572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1485.655745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1485.734385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.783455] LNet: 68802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1485.803815] LNetError: 68802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1485.808596] LNet: Removed LNI 192.168.204.33@tcp [ 1485.820083] LNet: Removed LNI 192.168.204.33@tcp1 [ 1485.860162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.963749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1486.005965] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1486.792688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1486.846961] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1488.191753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1488.273898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1488.408446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1488.475851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1488.539262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1488.588337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1488.653657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1488.709956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1488.754939] LNetError: 69075:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1488.991643] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1490.316181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1494.111598] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1494.126284] LNet: Skipped 1 previous similar message [ 1499.063450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1499.097182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1499.135140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1508.018320] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:59:33 (1787605173) [ 1508.103830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1508.143087] LNet: 69767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.157465] LNetError: 69767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.161944] LNetError: 69767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1508.185437] LNet: Removed LNI 192.168.204.33@tcp [ 1508.256768] LNet: Removed LNI 192.168.204.33@tcp1 [ 1508.315749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1508.416905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1508.458934] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1509.627635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1509.645859] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1511.025062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1511.190437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1511.256806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1511.356034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1511.451809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1511.552588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1511.645805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 0 [ 1511.915700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.480187] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.33@tcp [ 1512.488227] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1512.495559] LNet: Skipped 1 previous similar message [ 1527.864432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1528.047605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1528.091904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1528.136845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 500 [ 1528.352829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1535.394493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1535.507458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.531926] LNet: 70267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.541475] LNetError: 70267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.555289] LNetError: 70267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.564482] LNet: Removed LNI 192.168.204.33@tcp [ 1535.588841] LNet: Removed LNI 192.168.204.33@tcp1 [ 1535.670396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.769601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.826455] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1536.890964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1536.913890] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1538.060917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1538.178096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1538.311298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1538.357279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1538.396141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1538.432211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1538.466505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1538.504647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 0 [ 1539.808275] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1539.816038] LNet: Skipped 3 previous similar messages [ 1541.057937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1550.359714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1550.420878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1550.497151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1560.195477] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 17:00:25 (1787605225) [ 1560.934429] LNet: 71314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.946476] LNetError: 71314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.959829] LNetError: 71314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1560.972980] LNet: Removed LNI 192.168.204.33@tcp [ 1560.994477] LNet: Removed LNI 192.168.204.33@tcp1 [ 1561.799198] Key type .llcrypt unregistered [ 1561.803509] Key type ._llcrypt unregistered [ 1569.181658] Key type ._llcrypt registered [ 1569.190975] Key type .llcrypt registered [ 1569.427517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1569.470972] alg: No test for adler32 (adler32-zlib) [ 1570.382359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1570.411877] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1580.423909] Lustre: DEBUG MARKER: Initial discovery [ 1580.478563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1583.426752] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1583.513170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1583.585553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1583.614465] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1583.645739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1583.702121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.133@tcp -r 1 -m GET -l 3 [ 1583.712664] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1583.755406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.133@tcp -r 1 -m GET -e local_error [ 1583.791733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.133@tcp -r 1 -m PUT -l 6 [ 1585.722662] Lustre: DEBUG MARKER: Force oleg433-client.virtnet to discover 192.168.204.133@tcp (in background) [ 1585.789331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1587.248505] Lustre: DEBUG MARKER: Force oleg433-server to discover 192.168.204.33@tcp [ 1588.851388] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1590.003342] Lustre: DEBUG MARKER: Wait for 72074 [ 1592.069459] Lustre: DEBUG MARKER: Finished wait on 72074 [ 1592.151689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1592.210752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1592.217871] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1592.743692] LNet: 72536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1592.756258] LNetError: 72536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.840422] LNet: Removed LNI 192.168.204.33@tcp [ 1594.586493] Key type .llcrypt unregistered [ 1594.591527] Key type ._llcrypt unregistered [ 1615.586783] Key type ._llcrypt registered [ 1615.590695] Key type .llcrypt registered [ 1617.140732] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 17:01:22 (1787605282) [ 1618.097670] Key type .llcrypt unregistered [ 1618.103931] Key type ._llcrypt unregistered [ 1625.948880] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1626.202847] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1626.213730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1626.220653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1629.128343] Key type ._llcrypt registered [ 1629.133570] Key type .llcrypt registered [ 1629.452852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1629.468653] alg: No test for adler32 (adler32-zlib) [ 1630.519414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.597540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.635773] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1631.507425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1631.531912] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1639.973719] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 17:01:45 (1787605305) [ 1640.119241] LNet: 74624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.138232] LNetError: 74624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1640.145269] LNet: Removed LNI 10.1.2.3@tcp [ 1640.169939] LNet: Removed LNI 192.168.204.33@tcp [ 1641.179993] Key type .llcrypt unregistered [ 1641.186167] Key type ._llcrypt unregistered [ 1649.506191] Key type ._llcrypt registered [ 1649.507762] Key type .llcrypt registered [ 1649.867626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1649.892544] alg: No test for adler32 (adler32-zlib) [ 1650.924096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.056500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1651.115041] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1652.180488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1652.207807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1653.178797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.33@tcp [ 1654.240121] LNetError: Unexpected error -22 connecting to 192.168.204.33@tcp at host 192.168.204.33:988 [ 1661.048251] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 17:02:05 (1787605325) [ 1661.690671] LNet: 75770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.722130] LNetError: 75770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1661.747326] LNet: Removed LNI 10.1.2.3@tcp [ 1661.774112] LNet: Removed LNI 192.168.204.33@tcp [ 1663.127856] Key type .llcrypt unregistered [ 1663.132194] Key type ._llcrypt unregistered [ 1671.506972] Key type ._llcrypt registered [ 1671.508731] Key type .llcrypt registered [ 1671.738938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1671.754651] alg: No test for adler32 (adler32-zlib) [ 1672.801940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.940607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1672.988770] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1674.521052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1674.592662] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 1676.255863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.33@tcp1 --nid 192.168.204.33@tcp2 [ 1681.944490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp1 192.168.204.33@tcp2 [ 1681.961484] LNetError: 76611:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.33@tcp1 to 192.168.204.33@tcp2 - Route table may be misconfigured [ 1681.983631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp2 192.168.204.33@tcp1 [ 1690.158106] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 17:02:35 (1787605355) [ 1690.335734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.359983] LNet: 77186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1690.367751] LNetError: 77186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.423281] LNet: Removed LNI 192.168.204.33@tcp1 [ 1691.430357] LNetError: 77186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.464907] LNet: Removed LNI 192.168.204.33@tcp2 [ 1691.512591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.587711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.660804] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1692.543729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1692.578565] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1693.616603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1693.678729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.33@tcp [ 1693.730110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp1 192.168.204.33@tcp1 [ 1693.794539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp -d 192.168.204.33@tcp -e network_timeout [ 1693.830794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -e network_timeout [ 1693.869222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -e network_timeout [ 1693.910086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -e network_timeout [ 1693.944751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1693.964933] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1693.990961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1694.130133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1694.253146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1694.624244] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1702.140738] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 17:02:47 (1787605367) [ 1702.267044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.295100] LNet: 78042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.305570] LNetError: 78042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.318741] LNet: Removed LNI 192.168.204.33@tcp [ 1702.341663] LNet: Removed LNI 192.168.204.33@tcp1 [ 1702.369321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.504835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1703.592781] Key type .llcrypt unregistered [ 1703.597934] Key type ._llcrypt unregistered [ 1717.551742] Key type ._llcrypt registered [ 1717.555265] Key type .llcrypt registered [ 1719.204986] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 17:03:04 (1787605384) [ 1719.560668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1719.582254] alg: No test for adler32 (adler32-zlib) [ 1720.579703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.049954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1723.953661] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 17:03:09 (1787605389) [ 1723.999870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.013566] LNet: 79181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.029939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.086447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1724.121519] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1725.229143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1725.264949] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1726.845889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1726.943621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp1 [ 1727.006988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp1 [ 1735.892976] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 17:03:21 (1787605401) [ 1738.517190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg433-server [ 1741.350450] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 17:03:26 (1787605406) [ 1744.017932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg433-server [ 1746.394834] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 17:03:31 (1787605411) [ 1748.611116] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg433-server [ 1750.823484] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 17:03:36 (1787605416) [ 1752.774524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg433-server [ 1754.775526] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 17:03:40 (1787605420) [ 1756.964551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg433-server [ 1759.020848] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 17:03:44 (1787605424) [ 1761.559655] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg433-server [ 1764.381574] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 17:03:49 (1787605429) [ 1766.601968] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg433-server [ 1768.943349] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 17:03:54 (1787605434) [ 1771.199713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg433-server [ 1773.993654] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 17:03:58 (1787605438) [ 1776.506118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg433-server [ 1778.673811] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 17:04:04 (1787605444) [ 1778.880408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.916694] LNet: 83067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1778.923695] LNetError: 83067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.935214] LNet: Removed LNI 192.168.204.33@tcp [ 1779.942803] LNetError: 83067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.967090] LNet: Removed LNI 192.168.204.33@tcp1 [ 1780.062827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.176263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1780.233134] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1780.994871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1781.065396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1781.298256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.332672] LNet: 83215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.342036] LNetError: 83215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1781.374645] LNet: Removed LNI 192.168.204.33@tcp [ 1781.402354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.493126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1781.528320] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1782.398847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1783.981667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.051598] LNet: 83369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.060681] LNetError: 83369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.087648] LNet: Removed LNI 192.168.204.33@tcp [ 1784.124952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.243538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.294492] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1785.289182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1785.458597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1786.956477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.011245] LNet: Removed LNI 192.168.204.33@tcp [ 1788.041803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.129967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1788.177105] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1789.099348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1789.423637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.498159] LNet: 83671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1789.520034] LNet: 83671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1789.531750] LNetError: 83671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1789.538383] LNetError: 83671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1790.570964] LNet: Removed LNI 192.168.204.33@tcp [ 1790.646805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1790.752293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.819619] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1791.847249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1791.924618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1792.212711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.254057] LNet: Removed LNI 192.168.204.33@tcp [ 1793.307874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.445642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1793.520409] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1794.791625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1795.210442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.339715] LNet: Removed LNI 192.168.204.33@tcp [ 1796.404442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.488826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.525495] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1797.559859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1797.681210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1798.069951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.094197] LNet: 84112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.098535] LNet: 84112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1798.103749] LNetError: 84112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.111246] LNetError: 84112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1798.130737] LNet: Removed LNI 192.168.204.33@tcp [ 1798.173501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.259341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1798.287674] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1799.742715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1800.171990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.265106] LNet: Removed LNI 192.168.204.33@tcp [ 1801.334155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.446983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.516120] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1802.631878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1802.727636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1803.030453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.087540] LNet: Removed LNI 192.168.204.33@tcp [ 1804.154433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.228434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1804.245153] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1805.034857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1805.382215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.441908] LNet: Removed LNI 192.168.204.33@tcp [ 1806.484912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.589442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.629584] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1807.670714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1807.778490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1808.157960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.259057] LNet: Removed LNI 192.168.204.33@tcp [ 1809.337980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.442511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1809.493399] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1810.333682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1810.704644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.815214] LNet: Removed LNI 192.168.204.33@tcp [ 1811.864100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.979128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1812.028792] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1813.224784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1813.293791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.373111] LNet: Removed LNI 192.168.204.33@tcp [ 1813.441573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.506607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.539367] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1814.351420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1814.422138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.479523] LNet: 85113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.487413] LNet: 85113:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 1814.498334] LNetError: 85113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.511554] LNetError: 85113:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1814.529888] LNet: Removed LNI 192.168.204.33@tcp [ 1814.556643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.622505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1814.711065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.774440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.870302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1814.949844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.008224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.073205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1815.197994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.232983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.280899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1815.305847] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1816.117139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1816.179807] LNet: Removed LNI 192.168.204.33@tcp [ 1816.218526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.300320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1823.986187] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 17:04:49 (1787605489) [ 1824.128056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.228390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.326735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1824.355445] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1826.009756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.044133] LNet: Removed LNI 192.168.204.33@tcp231 [ 1826.072037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.156878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1826.196712] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1826.208645] LNet: 86012:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1826.883510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1826.926750] LNet: Removed LNI 192.168.204.33@tcp231 [ 1826.969629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1827.000076] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1828.223113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.304606] LNet: Removed LNI 192.168.204.33@tcp231 [ 1828.348294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.463710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1828.507586] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/0] [ 1828.512870] LNet: 86164:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1828.974472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1829.007675] LNet: Removed LNI 192.168.204.33@tcp231 [ 1829.029837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1829.052710] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/0] [ 1830.391772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.427768] LNet: Removed LNI 192.168.204.33@tcp231 [ 1830.449948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.518916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1830.577244] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/60] [ 1830.593554] LNet: 86315:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1830.919613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1830.945990] LNet: Removed LNI 192.168.204.33@tcp231 [ 1830.962413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1830.983936] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/60] [ 1832.247379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.280890] LNet: Removed LNI 192.168.204.33@tcp231 [ 1832.304898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.375245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1832.396267] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1832.891596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1832.954674] LNet: Removed LNI 192.168.204.33@tcp231 [ 1832.977109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1833.019879] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1841.542326] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 17:05:06 (1787605506) [ 1841.672872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.731801] LNet: Removed LNI 192.168.204.33@tcp231 [ 1841.782821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.847624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1841.884038] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1842.653618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1842.693851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1842.728763] LNet: Removed LNI 192.168.204.33@tcp232 [ 1842.750496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1842.805143] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1843.709749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1843.765143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1843.797127] LNet: Removed LNI 192.168.204.33@tcp232 [ 1843.824942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1843.854242] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1844.676374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1844.725489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1844.779612] LNet: Removed LNI 192.168.204.33@tcp232 [ 1844.810064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1844.832978] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1845.400819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1845.430348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1845.460550] LNet: Removed LNI 192.168.204.33@tcp232 [ 1845.515063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1854.220830] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 17:05:19 (1787605519) [ 1854.668924] LNet: 88307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.673974] LNet: 88307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1855.373125] Key type .llcrypt unregistered [ 1855.378953] Key type ._llcrypt unregistered [ 1864.375863] Key type ._llcrypt registered [ 1864.379275] Key type .llcrypt registered [ 1864.706844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1864.743456] alg: No test for adler32 (adler32-zlib) [ 1865.948156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1866.015254] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1873.560869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1884.547827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.594800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1884.650542] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1886.483597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 2 [ 1886.544428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 2 [ 1886.600943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 2 [ 1886.682932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 2 [ 1886.738513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_interrupt -r 2 [ 1886.814722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 2 [ 1886.875512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_interrupt -r 2 [ 1886.945223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 2 [ 1887.006811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1887.051851] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1887.063764] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1887.397628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1887.434666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1887.465753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1887.495706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1887.529420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1887.590666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1887.619637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1887.665586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1887.735609] LNet: 88975:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1887.785677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1887.849933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1887.884423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1887.920762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1887.959463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1887.998275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1888.698468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 2 [ 1888.753374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 2 [ 1888.805915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 2 [ 1888.856523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 2 [ 1888.929979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_dropped -r 2 [ 1889.006682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 2 [ 1889.077836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_dropped -r 2 [ 1889.144113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 2 [ 1889.213733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1889.264513] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1889.667209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1889.734593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1889.792823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1889.834851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1889.887297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1889.957294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1890.014450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1890.043910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1890.061775] LNet: 89096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1890.082536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1890.130435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1890.201881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1890.280420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1890.366786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1890.422711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1891.005881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 2 [ 1891.047453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 2 [ 1891.091473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 2 [ 1891.157629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 2 [ 1891.228994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_aborted -r 2 [ 1891.275971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 2 [ 1891.332179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_aborted -r 2 [ 1891.361884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 2 [ 1891.392701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1891.407698] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1891.672730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1891.713694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1891.741337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1891.778415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1891.808265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1891.841506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1891.878731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1891.928597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1891.957813] LNet: 89216:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1891.978743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1892.020536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1892.070910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1892.101465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1892.132402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1892.164773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1892.737618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 2 [ 1892.779813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 2 [ 1892.832114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 2 [ 1892.876759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 2 [ 1892.919104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_no_route -r 2 [ 1892.971052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 2 [ 1893.030618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_no_route -r 2 [ 1893.070615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 2 [ 1893.124519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1893.477372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1893.534687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1893.568332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1893.621677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1893.651429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1893.680341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1893.705840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1893.743469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1893.794682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1893.834138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1893.870674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1893.903628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1893.942920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1893.991795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1894.485869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 2 [ 1894.555445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 2 [ 1894.597386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 2 [ 1894.630949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 2 [ 1894.679927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_timeout -r 2 [ 1894.725843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 2 [ 1894.783355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_timeout -r 2 [ 1894.823491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 2 [ 1894.870914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1894.903481] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1894.909798] LNet: Skipped 1 previous similar message [ 1895.212548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1895.257789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1895.332689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1895.384878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1895.448291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1895.519803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1895.585898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1895.630912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1895.648900] LNet: 89457:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1895.653811] LNet: 89457:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1895.712881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1895.772334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1895.819635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1895.885247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1895.962592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1895.994611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1896.663190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 2 [ 1896.702511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 2 [ 1896.746568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 2 [ 1896.775648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 2 [ 1896.812256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_dropped -r 2 [ 1896.849508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 2 [ 1896.889712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_dropped -r 2 [ 1896.926813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 2 [ 1896.985176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1897.017768] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1897.233744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1897.266510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1897.304730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1897.336913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1897.373769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1897.400092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1897.445149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1897.487097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1897.530968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1897.567331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1897.616632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1897.652439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1897.692214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1897.739605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1903.585351] LNet: 89656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.598785] LNetError: 89656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.612880] LNet: Removed LNI 192.168.204.33@tcp [ 1903.625649] LNet: Removed LNI 192.168.204.33@tcp1 [ 1904.045683] Key type .llcrypt unregistered [ 1904.047567] Key type ._llcrypt unregistered [ 1921.738035] Key type ._llcrypt registered [ 1921.742634] Key type .llcrypt registered [ 1923.205492] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 17:06:28 (1787605588) [ 1923.750892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1923.779268] alg: No test for adler32 (adler32-zlib) [ 1924.728655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.800099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1924.829266] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1925.607935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1925.641364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1925.663288] LNetError: 90637:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1925.752538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.33@tcp [ 1925.863616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1932.721604] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 17:06:38 (1787605598) [ 1932.877436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.924930] LNet: 91222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.934794] LNetError: 91222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.957746] LNet: Removed LNI 192.168.204.33@tcp [ 1933.007094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.096891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1933.146429] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1934.330238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1934.374372] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1935.308891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.33@tcp [ 1935.342958] LNetError: 91493:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1936.160773] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1936.173706] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1936.659262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.33@tcp [ 1947.873347] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 17:06:52 (1787605612) [ 1950.631364] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg433-server [ 1953.279649] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 17:06:58 (1787605618) [ 1956.189623] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg433-server [ 1959.009241] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 17:07:04 (1787605624) [ 1962.510721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg433-server [ 1965.008781] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 17:07:10 (1787605630) [ 1965.110232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.136989] LNet: 93123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.142984] LNetError: 93123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.178072] LNet: Removed LNI 192.168.204.33@tcp [ 1966.191130] LNetError: 93123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.237431] LNet: Removed LNI 192.168.204.33@tcp1 [ 1966.314091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.445120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.488506] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1967.623288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1967.736303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1967.833506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1967.986758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1968.056739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1968.177422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1976.513665] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 17:07:21 (1787605641) [ 1976.571955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1976.605432] LNet: 93851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.617544] LNetError: 93851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.646793] LNet: Removed LNI 192.168.204.33@tcp [ 1976.681489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.759674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.788858] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1984.850503] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 17:07:30 (1787605650) [ 1985.032607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.071327] LNet: 94557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.099481] LNetError: 94557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.121444] LNet: Removed LNI 192.168.204.33@tcp [ 1985.184972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.289967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.318248] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1993.721768] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 17:07:38 (1787605658) [ 1995.362827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1997.646931] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 17:07:42 (1787605662) [ 1998.360443] LNet: 95596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.368963] LNetError: 95596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.393679] LNet: Removed LNI 192.168.204.33@tcp [ 1999.308176] Key type .llcrypt unregistered [ 1999.310293] Key type ._llcrypt unregistered [ 2006.446597] Key type ._llcrypt registered [ 2006.450809] Key type .llcrypt registered [ 2006.829211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2006.857582] alg: No test for adler32 (adler32-zlib) [ 2007.910105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2007.957705] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2015.416897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2020.112331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.133@tcp [ 2021.672171] LNet: 96023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.681381] LNetError: 96023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.692447] LNet: Removed LNI 192.168.204.33@tcp [ 2022.171457] Key type .llcrypt unregistered [ 2022.174194] Key type ._llcrypt unregistered [ 2033.857233] Key type ._llcrypt registered [ 2033.859855] Key type .llcrypt registered [ 2035.428647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2037.472582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2039.461402] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 17:08:24 (1787605704) [ 2041.483448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2043.734829] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 17:08:29 (1787605709) [ 2045.768397] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg433-server [ 2047.705782] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 17:08:33 (1787605713) [ 2050.708992] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg433-server [ 2053.517448] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 17:08:38 (1787605718) [ 2055.596435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.621177] alg: No test for adler32 (adler32-zlib) [ 2056.910607] Lustre: Lustre: Build Version: 2.17.57_80_ged04d84 [ 2057.241811] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2058.935219] Key type lgssc registered [ 2060.304656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2085.139548] Key type lgssc unregistered [ 2085.402872] LNet: 99203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.407906] LNetError: 99203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.419795] LNet: Removed LNI 192.168.204.33@tcp [ 2086.123479] Key type .llcrypt unregistered [ 2086.125953] Key type ._llcrypt unregistered [ 2101.736929] Key type ._llcrypt registered [ 2101.743482] Key type .llcrypt registered [ 2103.281473] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 17:09:28 (1787605768) [ 2103.733693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2103.774448] alg: No test for adler32 (adler32-zlib) [ 2104.873761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2104.979825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2105.053050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2105.146762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2105.250464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2112.901417] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 17:09:38 (1787605778) [ 2113.077986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.113846] LNet: 100619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.179482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.336701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.376475] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2114.335511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.391182] LNet: 100766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2114.412418] LNetError: 100766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2114.438498] LNet: Removed LNI 192.168.204.33@tcp [ 2114.514488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.640762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2114.682526] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2115.732700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.802283] LNet: 100794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.816669] LNetError: 100794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.857708] LNet: Removed LNI 192.168.204.33@tcp [ 2115.915378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.055529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2116.111050] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2124.180569] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 17:09:49 (1787605789) [ 2124.610672] LNet: 101386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2124.632984] LNetError: 101386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2124.649288] LNet: Removed LNI 192.168.204.33@tcp [ 2125.447500] Key type .llcrypt unregistered [ 2125.449276] Key type ._llcrypt unregistered [ 2133.467185] Key type ._llcrypt registered [ 2133.469232] Key type .llcrypt registered [ 2134.020482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2134.091329] alg: No test for adler32 (adler32-zlib) [ 2135.291335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.402312] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2135.408842] LNet: *** cfs_fail_loc=e003, val=100*** [ 2136.703702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2136.732558] LNet: *** cfs_fail_loc=e003, val=100*** [ 2136.736211] LNet: Skipped 2 previous similar messages [ 2136.748689] Ping reply received with mapping: [ 2136.756910] found metadata reply NI[192.168.204.33@tcp] [ 2136.763837] 21 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2136.776316] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2136.786039] 5a 5a 5a 5a 5a 5a 5a 5a [ 2137.038216] LNet: 101788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.046223] LNetError: 101788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.091014] LNet: Removed LNI 192.168.204.33@tcp [ 2138.331584] Key type .llcrypt unregistered [ 2138.337911] Key type ._llcrypt unregistered [ 2153.911897] Key type ._llcrypt registered [ 2153.919489] Key type .llcrypt registered [ 2155.403518] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 17:10:20 (1787605820) [ 2155.953275] Key type .llcrypt unregistered [ 2155.957140] Key type ._llcrypt unregistered [ 2165.278844] Key type ._llcrypt registered [ 2165.282542] Key type .llcrypt registered [ 2165.672643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2165.710702] alg: No test for adler32 (adler32-zlib) [ 2167.153817] LNetError: 102846:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2168.066500] Key type .llcrypt unregistered [ 2168.068344] Key type ._llcrypt unregistered [ 2173.879165] Key type ._llcrypt registered [ 2173.882855] Key type .llcrypt registered [ 2175.789497] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 17:10:40 (1787605840) [ 2177.030266] Key type .llcrypt unregistered [ 2177.032031] Key type ._llcrypt unregistered [ 2184.174790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2184.181524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2185.773617] Key type ._llcrypt registered [ 2185.779227] Key type .llcrypt registered [ 2186.038741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2186.062435] alg: No test for adler32 (adler32-zlib) [ 2187.075129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2187.166788] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2198.118822] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 17:11:03 (1787605863) [ 2198.224499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2198.255112] LNet: 104620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2198.261684] LNetError: 104620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2198.284154] LNet: Removed LNI 10.1.2.3@tcp [ 2198.321285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.111816] Key type .llcrypt unregistered [ 2199.114531] Key type ._llcrypt unregistered [ 2206.067851] Key type ._llcrypt registered [ 2206.079315] Key type .llcrypt registered [ 2206.440774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2206.466342] alg: No test for adler32 (adler32-zlib) [ 2207.501079] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2207.605693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2207.896523] LNet: 104920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2207.904322] LNetError: 104920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2207.945901] LNet: Removed LNI 192.168.204.33@tcp [ 2208.940546] Key type .llcrypt unregistered [ 2208.942977] Key type ._llcrypt unregistered [ 2222.914227] Key type ._llcrypt registered [ 2222.915762] Key type .llcrypt registered [ 2224.264277] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 17:11:29 (1787605889) [ 2225.237723] Key type .llcrypt unregistered [ 2225.246405] Key type ._llcrypt unregistered [ 2231.927342] Key type ._llcrypt registered [ 2231.929764] Key type .llcrypt registered [ 2232.141340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2232.160304] alg: No test for adler32 (adler32-zlib) [ 2240.939793] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 17:11:46 (1787605906) [ 2241.081811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.144283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.221387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2241.255903] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2242.188790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2242.299247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2242.357584] LNet: Removed LNI 192.168.204.33@tcp [ 2242.852802] LNet: 106694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.522133] Key type .llcrypt unregistered [ 2243.525168] Key type ._llcrypt unregistered [ 2257.028739] Key type ._llcrypt registered [ 2257.032575] Key type .llcrypt registered [ 2258.227521] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 17:12:03 (1787605923) [ 2258.758161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2258.788788] alg: No test for adler32 (adler32-zlib) [ 2259.800666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.894429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2259.923906] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2260.612904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2260.654904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.33@tcp [ 2260.689332] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2260.692616] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2260.695083] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2260, 0, 0 [ 2260.699183] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2260.702470] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2260.706164] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2260.708815] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2260, 0, 0 [ 2260.714455] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2260.718701] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2260.722078] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2260.724832] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2260, 0, 2310 [ 2260.728624] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2260.730880] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2260.734669] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2260.739131] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2260, 0, 2310 [ 2260.746211] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2268.898432] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 17:12:13 (1787605933) [ 2269.435361] LNet: 108235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2269.440343] LNetError: 108235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2269.454784] LNet: Removed LNI 192.168.204.33@tcp [ 2270.092430] Key type .llcrypt unregistered [ 2270.096660] Key type ._llcrypt unregistered [ 2278.425356] Key type ._llcrypt registered [ 2278.429704] Key type .llcrypt registered [ 2278.789305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2278.822201] alg: No test for adler32 (adler32-zlib) [ 2279.773768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2279.820654] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2290.352935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2302.429703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.514783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2302.562759] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 2303.431315] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2303.556454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2303.572487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2303.590380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2306.149216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2306.190221] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2321.805859] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 17:13:06 (1787605986) [ 2322.261758] LNet: 109749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.274979] LNetError: 109749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.292478] LNet: Removed LNI 192.168.204.33@tcp [ 2322.294656] LNet: Removed LNI 192.168.204.33@tcp1 [ 2322.344041] LNet: Removed LNI 10.1.2.3@tcp99 [ 2323.418676] Key type .llcrypt unregistered [ 2323.420707] Key type ._llcrypt unregistered [ 2332.169505] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2332.177836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2335.079714] Key type ._llcrypt registered [ 2335.084747] Key type .llcrypt registered [ 2335.442941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2335.475700] alg: No test for adler32 (adler32-zlib) [ 2336.587756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.653356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2336.717640] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2337.933981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2337.966081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2341.814493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2341.865759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2341.912682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2341.930922] LNetError: 110499:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2341.968914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2342.040291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2342.797087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2342.898572] LNet: 110519:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.923141] LNetError: 110519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2342.934885] LNet: Removed LNI 10.1.2.3@tcp [ 2343.976769] LNet: Removed LNI 192.168.204.33@tcp [ 2345.078488] Key type .llcrypt unregistered [ 2345.080247] Key type ._llcrypt unregistered [ 2360.939909] Key type ._llcrypt registered [ 2360.941989] Key type .llcrypt registered [ 2362.755242] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 17:13:47 (1787606027) [ 2363.195228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2363.212173] alg: No test for adler32 (adler32-zlib) [ 2364.208823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.301454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.350978] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2365.163855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2365.246472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg433-client@tcp [ 2372.935944] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 17:13:58 (1787606038) [ 2373.078671] LNet: 112059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.088901] LNetError: 112059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.112448] LNet: Removed LNI 192.168.204.33@tcp [ 2374.006423] Key type .llcrypt unregistered [ 2374.010910] Key type ._llcrypt unregistered [ 2382.120092] Key type ._llcrypt registered [ 2382.124255] Key type .llcrypt registered [ 2382.534462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2382.605948] alg: No test for adler32 (adler32-zlib) [ 2383.715661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2383.779792] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2383.856080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2393.387269] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 17:14:18 (1787606058) [ 2395.769339] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2398.409162] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 17:14:23 (1787606063) [ 2400.232547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2402.637755] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 17:14:27 (1787606067) [ 2402.955914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.003482] LNet: 113494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2403.014436] LNetError: 113494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2403.038305] LNet: Removed LNI 192.168.204.33@tcp [ 2403.066109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.185441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2403.234890] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2404.253891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2404.331813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2413.294610] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 17:14:37 (1787606077) [ 2413.391703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.434281] LNet: 114197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.458600] LNetError: 114197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.476497] LNet: Removed LNI 192.168.204.33@tcp [ 2413.523173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.612284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2413.645114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2414.503569] Key type .llcrypt unregistered [ 2414.508358] Key type ._llcrypt unregistered [ 2430.976891] Key type ._llcrypt registered [ 2430.983397] Key type .llcrypt registered [ 2432.935338] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 17:14:57 (1787606097) [ 2433.305367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2433.325353] alg: No test for adler32 (adler32-zlib) [ 2434.291324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2434.412386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2434.447982] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2435.603589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2435.698140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2443.077746] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 17:15:08 (1787606108) [ 2443.250994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2443.291581] LNet: 115763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.318474] LNetError: 115763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.367731] LNet: Removed LNI 192.168.204.33@tcp [ 2443.432933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.503234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2443.528436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2450.962086] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 17:15:16 (1787606116) [ 2451.240398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2451.287175] LNet: 116347:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2451.345925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.468620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2452.166444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2452.198517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2452.267330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2453.060497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2453.118731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2453.257727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2453.314094] LNet: 116380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.362393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.514457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2454.359264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2454.427272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2454.525803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2455.145942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2455.224643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2463.787508] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 17:15:28 (1787606128) [ 2464.252391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2464.298927] LNet: 116982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.346176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.484636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2464.522907] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2465.773590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2465.828974] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2467.104761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2467.237307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2467.554027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.576973] LNetError: 117261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.586559] LNet: Removed LNI 192.168.204.33@tcp [ 2467.612309] LNet: Removed LNI 192.168.204.33@tcp2 [ 2467.665371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2467.777705] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2467.793525] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2467.803766] LNet: 117267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2468.538369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2468.757473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2468.784720] LNet: 117304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.789966] LNet: 117304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2468.794960] LNetError: 117304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2468.801320] LNetError: 117304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2468.806560] LNet: Removed LNI 192.168.204.33@tcp [ 2468.821411] LNet: Removed LNI 192.168.204.33@tcp2 [ 2468.858925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.917831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2468.935505] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2470.108518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2470.131131] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2471.500866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2471.562343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2471.684293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2472.119251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2472.146943] LNetError: 117589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.153025] LNetError: 117589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2472.167572] LNet: Removed LNI 192.168.204.33@tcp [ 2472.197202] LNet: Removed LNI 192.168.204.33@tcp2 [ 2472.245395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.362348] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2472.372991] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2472.382780] LNet: 117595:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2473.540321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2473.922872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2473.959646] LNet: Removed LNI 192.168.204.33@tcp [ 2474.981185] LNet: Removed LNI 192.168.204.33@tcp2 [ 2475.062649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2475.208251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2475.319662] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2476.935225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2477.000060] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2478.413278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2478.482895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2478.903812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2478.930990] LNet: 117912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2478.939405] LNet: 117912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2478.948920] LNetError: 117912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2478.956669] LNetError: 117912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2478.974163] LNet: Removed LNI 192.168.204.33@tcp [ 2478.992212] LNet: Removed LNI 192.168.204.33@tcp2 [ 2479.049780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.209156] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2479.231654] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2479.247449] LNet: 117918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2479.662668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2479.973809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.055283] LNet: Removed LNI 192.168.204.33@tcp [ 2481.073953] LNet: Removed LNI 192.168.204.33@tcp2 [ 2481.118324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.195614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2481.230711] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2482.317292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2482.354612] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2483.504460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2483.625323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2483.725054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2484.094455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.152119] LNet: Removed LNI 192.168.204.33@tcp [ 2486.196161] LNet: Removed LNI 192.168.204.33@tcp2 [ 2486.256849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.335220] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2486.342509] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2486.358451] LNet: 118247:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.141520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2487.427722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2487.455075] LNetError: 118284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.465244] LNetError: 118284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2487.476815] LNet: Removed LNI 192.168.204.33@tcp [ 2488.551406] LNet: Removed LNI 192.168.204.33@tcp2 [ 2488.580966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2488.724355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2488.769382] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2489.838638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2489.880366] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2491.151249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2491.202602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2491.286621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2491.904169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2492.959291] LNet: Removed LNI 192.168.204.33@tcp [ 2492.985644] LNet: Removed LNI 192.168.204.33@tcp2 [ 2493.055341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2493.181732] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2493.197592] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2493.208231] LNet: 118587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2493.951810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2494.413765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2495.455678] LNet: Removed LNI 192.168.204.33@tcp [ 2495.480495] LNet: Removed LNI 192.168.204.33@tcp2 [ 2495.535676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.634392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2495.675352] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2496.861639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2496.894896] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2498.336565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2498.379525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2498.445605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2498.541105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2499.012701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2499.036797] LNet: 118941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.040617] LNet: 118941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2500.063174] LNet: Removed LNI 192.168.204.33@tcp [ 2501.094504] LNet: Removed LNI 192.168.204.33@tcp2 [ 2501.143604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.269359] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2501.278689] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2501.300063] LNet: 118947:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2502.109284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2510.620420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2512.762399] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 17:16:18 (1787606178) [ 2513.118483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2513.182955] LNetError: 119708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.194531] LNetError: 119708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2513.207673] LNet: Removed LNI 192.168.204.33@tcp [ 2514.279289] LNet: Removed LNI 192.168.204.33@tcp2 [ 2514.310338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.609847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2514.630338] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2515.478763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2515.513365] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2516.447850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2516.469034] LNet: Removed LNI 10.1.2.3@tcp [ 2516.480685] LNet: Removed LNI 192.168.204.33@tcp [ 2516.498775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2516.593612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2516.621170] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2516.635925] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2517.183474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2517.218080] LNet: Removed LNI 192.168.204.33@tcp [ 2517.241594] LNet: Removed LNI 10.1.2.3@tcp [ 2525.262219] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 17:16:30 (1787606190) [ 2525.823470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2525.887431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2526.002377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2526.053372] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2526.933038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2526.987373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.048434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.102855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.147963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.211252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.264252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.313237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.364414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.428329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.533475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2527.885041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2537.425543] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 17:16:42 (1787606202) [ 2537.865484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2537.915987] LNet: 121323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.925752] LNet: 121323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2538.984077] LNet: Removed LNI 192.168.204.33@tcp [ 2539.088890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2539.397986] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2539.697848] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2539.710115] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2539.728226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2541.257887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2541.299402] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2542.794849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2542.879887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2545.977731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2546.030721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2546.118196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.33@tcp -m PUT -e local_timeout -r 1 [ 2546.181748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2546.235389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2546.238060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2546.479922] LNetError: 121381:0:(lib-move.c:4801:lnet_parse()) 192.168.204.33@tcp, src 192.168.204.33@tcp: Dropping PUT (error -108 looking up sender) [ 2547.491282] LNetError: 121845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2547.505143] LNetError: 121845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2547.518930] LNet: Removed LNI 10.1.2.3@tcp [ 2548.582851] LNet: Removed LNI 192.168.204.33@tcp [ 2549.173809] Key type .llcrypt unregistered [ 2549.177823] Key type ._llcrypt unregistered [ 2565.483118] Key type ._llcrypt registered [ 2565.484907] Key type .llcrypt registered [ 2568.287993] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 17:17:12 (1787606232) [ 2569.653354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.720831] alg: No test for adler32 (adler32-zlib) [ 2570.981975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2571.861566] LNet: 122700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.150395] Key type .llcrypt unregistered [ 2573.156677] Key type ._llcrypt unregistered [ 2583.065047] Key type ._llcrypt registered [ 2583.068898] Key type .llcrypt registered [ 2583.276301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2583.299478] alg: No test for adler32 (adler32-zlib) [ 2584.478239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2584.565989] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2592.427281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2592.901135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2593.726788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.133@tcp [ 2593.825663] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 2594.136200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2594.197878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2594.273690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2594.370515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2594.414959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2594.476929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2594.541416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2594.576924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2594.590438] LNet: 123155:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2594.604658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2594.632208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2594.669194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2594.697431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2594.733457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2594.754798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2595.651560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2596.114336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2596.162633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2596.235737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2596.302346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2596.377658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2596.448193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2596.536616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2596.631626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2596.680664] LNet: 123273:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2596.734888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2596.813994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2596.871421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2596.923389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2597.006429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2597.101601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2598.153716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.133@tcp [ 2598.677192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2598.730287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2598.775234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2598.809430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2598.858979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2598.943826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2599.007970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2599.063326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2599.102148] LNet: 123390:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2599.135746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2599.197214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2599.297689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2599.376643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2599.472833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2599.538260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2600.614141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2600.966891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2601.027289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2601.095613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2601.164073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2601.228895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2601.273859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2601.310098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2601.395331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2601.434641] LNet: 123508:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2601.474072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2601.507630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2601.570981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2601.633792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2601.728123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2601.772187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2602.419251] LNet: 123558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2602.439511] LNetError: 123558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2602.463073] LNet: Removed LNI 192.168.204.33@tcp [ 2603.161964] Key type .llcrypt unregistered [ 2603.166499] Key type ._llcrypt unregistered [ 2627.155086] Key type ._llcrypt registered [ 2627.159973] Key type .llcrypt registered [ 2629.244765] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 17:18:13 (1787606293) [ 2629.618935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2629.639880] alg: No test for adler32 (adler32-zlib) [ 2630.665654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2631.156744] LNet: 124448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2631.914422] Key type .llcrypt unregistered [ 2631.917923] Key type ._llcrypt unregistered [ 2640.238978] Key type ._llcrypt registered [ 2640.242604] Key type .llcrypt registered [ 2640.508664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2640.544403] alg: No test for adler32 (adler32-zlib) [ 2641.648596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2641.689841] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2651.316360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2662.940082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2663.041800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2663.096055] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 2664.554365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2665.344188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.133@tcp [ 2665.606327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2665.666140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2665.717742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2665.755534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2665.790196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2665.833046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2665.864621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2665.928221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2665.963693] LNet: 125106:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2665.983939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2666.027941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2666.088304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2666.127888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2666.177241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2666.229704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2667.123680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2667.295743] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.133@tcp, 192.168.204.133@tcp1 [ 2667.442832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2667.486666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2667.552464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2667.600476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2667.662760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2667.725965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2667.759370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2667.827753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2667.861179] LNet: 125224:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2667.902866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2667.972286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2668.016828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2668.049896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2668.085911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2668.145063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2669.032699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.133@tcp [ 2669.367588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2669.411500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2669.504362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2669.559662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2669.608871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2669.655856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2669.730337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2669.799636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2669.816324] LNet: 125341:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2669.841382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2669.883516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2669.925984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2669.994880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2670.050907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2670.080305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2670.751560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2671.140524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2671.181568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2671.254757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2671.290407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2671.325163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2671.366962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2671.412346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2671.460833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2671.509553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2671.562358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2671.604939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2671.645296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2671.703308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2671.754711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2677.447984] LNet: 125544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2677.459199] LNetError: 125544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2677.475688] LNet: Removed LNI 192.168.204.33@tcp [ 2677.506455] LNet: Removed LNI 192.168.204.33@tcp1 [ 2678.030430] Key type .llcrypt unregistered [ 2678.032766] Key type ._llcrypt unregistered [ 2699.171334] Key type ._llcrypt registered [ 2699.174765] Key type .llcrypt registered [ 2701.056528] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 17:19:25 (1787606365) [ 2701.923064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2701.949591] alg: No test for adler32 (adler32-zlib) [ 2702.982303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2703.019518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2703.035725] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2703.801391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2703.849575] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2704.867513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2705.028616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2706.079217] LNet: Removed LNI 192.168.204.33@tcp [ 2716.132478] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 17:19:40 (1787606380) [ 2718.256432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2720.559562] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 17:19:45 (1787606385) [ 2729.890691] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 17:19:55 (1787606395) [ 2738.819236] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 17:20:04 (1787606404) [ 2747.574690] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 17:20:12 (1787606412) [ 2756.505527] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 17:20:21 (1787606421) [ 2765.874591] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 17:20:31 (1787606431) [ 2774.632240] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 17:20:39 (1787606439) [ 2783.296591] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 17:20:49 (1787606449) [ 2792.640688] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 17:20:57 (1787606457) [ 2800.932140] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 17:21:06 (1787606466) [ 2809.069600] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 17:21:14 (1787606474) [ 2817.926854] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 17:21:22 (1787606482) [ 2827.694541] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 17:21:32 (1787606492) [ 2836.380389] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 17:21:41 (1787606501) [ 2846.939892] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 17:21:52 (1787606512) [ 2856.094821] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 17:22:01 (1787606521) [ 2865.350708] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 17:22:10 (1787606530) [ 2875.556853] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 17:22:20 (1787606540) [ 2875.733772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2875.776819] LNet: 137243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2875.794099] LNetError: 137243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2875.843526] LNet: Removed LNI 192.168.204.33@tcp2 [ 2875.937312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2876.080761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2876.109624] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2877.950667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2877.980319] LNet: 137393:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2877.994317] LNetError: 137393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2878.010909] LNet: Removed LNI 192.168.204.33@tcp [ 2878.037214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2878.171888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2878.198353] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2887.241726] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 17:22:32 (1787606552) [ 2900.488270] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 17:22:45 (1787606565) [ 2910.015365] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 17:22:55 (1787606575) [ 2919.570461] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 17:23:04 (1787606584) [ 2929.541333] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 17:23:14 (1787606594) [ 2938.493257] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 17:23:23 (1787606603) [ 2946.839843] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 17:23:32 (1787606612) [ 2957.166941] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 17:23:42 (1787606622) [ 2966.075981] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 17:23:51 (1787606631) [ 2974.686037] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 17:24:00 (1787606640) [ 2982.950753] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 17:24:08 (1787606648) [ 2991.204674] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 17:24:16 (1787606656) [ 2999.633142] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2729 sec ========== 17:24:24 (1787606664) [ 2999.982601] LNet: 144866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2999.993652] LNetError: 144866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3000.022672] LNet: Removed LNI 192.168.204.33@tcp [ 3000.831784] Key type .llcrypt unregistered [ 3000.833834] Key type ._llcrypt unregistered [ 3013.766471] Key type ._llcrypt registered [ 3013.786168] Key type .llcrypt registered [ 3014.385761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3014.414866] alg: No test for adler32 (adler32-zlib) [ 3015.545860] Lustre: Lustre: Build Version: 2.17.57_80_ged04d84 [ 3015.863820] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 3017.567515] Key type lgssc registered [ 3018.862993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3088.722607] Lustre: Mounted lustre-client - version 2.17.57_80_ged04d84 [ 3096.304828] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3105.502699] Lustre: Unmounted lustre-client [ 3134.377582] Key type lgssc unregistered [ 3134.718833] LNet: 147830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3134.727793] LNetError: 147830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3134.746431] LNet: Removed LNI 192.168.204.33@tcp [ 3135.725461] Key type .llcrypt unregistered [ 3135.735583] Key type ._llcrypt unregistered