[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 424156129 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002341] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006834] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008009] pid_max: default: 32768 minimum: 301 [ 0.009120] LSM: Security Framework initializing [ 0.010041] Yama: becoming mindful. [ 0.011027] SELinux: Initializing. [ 0.013011] *** VALIDATE selinux *** [ 0.019710] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023693] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024117] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025088] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026108] *** VALIDATE tmpfs *** [ 0.027321] *** VALIDATE proc *** [ 0.029058] *** VALIDATE cgroup *** [ 0.030007] *** VALIDATE cgroup2 *** [ 0.031281] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034025] Spectre V2 : User space: Vulnerable [ 0.035006] Speculative Store Bypass: Vulnerable [ 0.038339] debug: unmapping init [mem 0xffffffff93259000-0xffffffff93260fff] [ 0.040168] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041620] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042020] ... version: 2 [ 0.043013] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046012] ... max period: 00007fffffffffff [ 0.047012] ... fixed-purpose events: 3 [ 0.048011] ... event mask: 000000070000000f [ 0.049279] rcu: Hierarchical SRCU implementation. [ 0.051359] smp: Bringing up secondary CPUs ... [ 0.052552] x86: Booting SMP configuration: [ 0.053021] .... node #0, CPUs: #1 #2 #3 [ 0.056370] smp: Brought up 1 node, 4 CPUs [ 0.058012] smpboot: Max logical packages: 1 [ 0.059019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.142633] node 0 deferred pages initialised in 81ms [ 0.146118] devtmpfs: initialized [ 0.148283] x86/mm: Memory block size: 128MB [ 0.152282] gcov: version magic: 0x41383552 [ 0.154161] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158123] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160323] pinctrl core: initialized pinctrl subsystem [ 0.163191] [ 0.163843] ************************************************************* [ 0.166011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168009] ** ** [ 0.170012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172009] ** ** [ 0.174011] ** This means that this kernel is built to expose internal ** [ 0.177012] ** IOMMU data structures, which may compromise security on ** [ 0.179014] ** your system. ** [ 0.181009] ** ** [ 0.183011] ** If you see this message and you are not debugging the ** [ 0.186013] ** kernel, report this immediately to your vendor! ** [ 0.188010] ** ** [ 0.190013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.193012] ************************************************************* [ 0.195719] NET: Registered protocol family 16 [ 0.197508] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.203054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.207030] cpuidle: using governor menu [ 0.208656] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.210533] PCI: Using configuration type 1 for base access [ 0.212130] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218101] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220015] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223073] cryptd: max_cpu_qlen set to 1000 [ 0.227264] ACPI: Added _OSI(Module Device) [ 0.229012] ACPI: Added _OSI(Processor Device) [ 0.230008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235488] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.240371] ACPI: Interpreter enabled [ 0.242052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.243013] ACPI: Using IOAPIC for interrupt routing [ 0.245102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248366] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.257952] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.260037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.262017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.265062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.270250] acpiphp: Slot [2] registered [ 0.271128] acpiphp: Slot [5] registered [ 0.272131] acpiphp: Slot [6] registered [ 0.274117] acpiphp: Slot [3] registered [ 0.275065] acpiphp: Slot [4] registered [ 0.276055] acpiphp: Slot [7] registered [ 0.277074] acpiphp: Slot [8] registered [ 0.278063] acpiphp: Slot [9] registered [ 0.280111] acpiphp: Slot [10] registered [ 0.281093] acpiphp: Slot [11] registered [ 0.283091] acpiphp: Slot [12] registered [ 0.284103] acpiphp: Slot [13] registered [ 0.286110] acpiphp: Slot [14] registered [ 0.287083] acpiphp: Slot [15] registered [ 0.288092] acpiphp: Slot [16] registered [ 0.289069] acpiphp: Slot [17] registered [ 0.290023] acpiphp: Slot [18] registered [ 0.291000] acpiphp: Slot [19] registered [ 0.293071] acpiphp: Slot [20] registered [ 0.294067] acpiphp: Slot [21] registered [ 0.295088] acpiphp: Slot [22] registered [ 0.297107] acpiphp: Slot [23] registered [ 0.298115] acpiphp: Slot [24] registered [ 0.300095] acpiphp: Slot [25] registered [ 0.301085] acpiphp: Slot [26] registered [ 0.303124] acpiphp: Slot [27] registered [ 0.304092] acpiphp: Slot [28] registered [ 0.305081] acpiphp: Slot [29] registered [ 0.307107] acpiphp: Slot [30] registered [ 0.308089] acpiphp: Slot [31] registered [ 0.309057] PCI host bridge to bus 0000:00 [ 0.311016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.323025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.325179] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.329052] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.332705] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.340013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.345058] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.348023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.350015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.353016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.355422] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.357781] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.360051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.363826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.366000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.375021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385859] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.391010] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.400014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413786] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.421888] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.427012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.431013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.445909] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.458185] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.460362] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.463374] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.466425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.468210] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.471103] iommu: Default domain type: Passthrough [ 0.473388] SCSI subsystem initialized [ 0.474096] ACPI: bus type USB registered [ 0.476100] usbcore: registered new interface driver usbfs [ 0.477067] usbcore: registered new interface driver hub [ 0.479112] usbcore: registered new device driver usb [ 0.481168] pps_core: LinuxPPS API ver. 1 registered [ 0.484012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.488062] PTP clock support registered [ 0.490137] EDAC MC: Ver: 3.0.0 [ 0.492545] PCI: Using ACPI for IRQ routing [ 0.494756] NetLabel: Initializing [ 0.496007] NetLabel: domain hash size = 128 [ 0.497009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.499079] NetLabel: unlabeled traffic allowed by default [ 0.501111] vgaarb: loaded [ 0.503256] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.504012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.513064] clocksource: Switched to clocksource kvm-clock [ 0.617104] VFS: Disk quotas dquot_6.6.0 [ 0.618802] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.621575] *** VALIDATE ramfs *** [ 0.622945] *** VALIDATE hugetlbfs *** [ 0.624683] pnp: PnP ACPI init [ 0.627441] pnp: PnP ACPI: found 6 devices [ 0.652137] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.654597] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.656140] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.657605] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.659813] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.662375] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.665251] NET: Registered protocol family 2 [ 0.667881] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.673126] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.677522] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.683499] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.687358] TCP: Hash tables configured (established 65536 bind 65536) [ 0.690508] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.693616] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.696451] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.699585] NET: Registered protocol family 1 [ 0.702145] RPC: Registered named UNIX socket transport module. [ 0.704839] RPC: Registered udp transport module. [ 0.706759] RPC: Registered tcp transport module. [ 0.709123] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.712505] NET: Registered protocol family 44 [ 0.714386] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.717037] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.719647] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.722109] PCI: CLS 0 bytes, default 64 [ 0.724184] Unpacking initramfs... [ 2.101367] debug: unmapping init [mem 0xffff9fc27cc64000-0xffff9fc27ffcffff] [ 2.105183] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.107780] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.110668] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.609280] Initialise system trusted keyrings [ 2.611239] Key type blacklist registered [ 2.613471] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.626476] zbud: loaded [ 2.630075] *** VALIDATE nfs *** [ 2.631309] *** VALIDATE nfs4 *** [ 2.632974] pstore: using deflate compression [ 2.637328] Platform Keyring initialized [ 2.750572] NET: Registered protocol family 38 [ 2.753182] Key type asymmetric registered [ 2.754824] Asymmetric key parser 'x509' registered [ 2.756989] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.760458] io scheduler mq-deadline registered [ 2.762515] io scheduler kyber registered [ 2.764451] io scheduler bfq registered [ 2.767194] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.770642] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.774173] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.777578] ACPI: Power Button [PWRF] [ 2.784077] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.791981] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.804891] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.836186] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.866295] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.872044] Non-volatile memory driver v1.3 [ 2.873510] Linux agpgart interface v0.103 [ 2.908542] virtio_blk virtio1: [vda] 147232 512-byte logical blocks (75.4 MB/71.9 MiB) [ 2.911851] vda: detected capacity change from 0 to 75382784 [ 2.926077] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.929602] vdb: detected capacity change from 0 to 1073741824 [ 2.937411] libphy: Fixed MDIO Bus: probed [ 2.942488] usbcore: registered new interface driver usbserial_generic [ 2.945851] usbserial: USB Serial support registered for generic [ 2.948541] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.953500] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.955595] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.958437] mousedev: PS/2 mouse device common for all mice [ 2.961693] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.963696] rtc_cmos 00:05: RTC can wake from S4 [ 2.969581] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.970605] rtc_cmos 00:05: registered as rtc0 [ 2.975762] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.978258] intel_pstate: CPU model not supported [ 2.978981] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.985701] hid: raw HID events driver (C) Jiri Kosina [ 2.988303] usbcore: registered new interface driver usbhid [ 2.990566] usbhid: USB HID core driver [ 2.992623] drop_monitor: Initializing network drop monitor service [ 2.994962] Initializing XFRM netlink socket [ 2.997732] NET: Registered protocol family 10 [ 3.000809] Segment Routing with IPv6 [ 3.002722] NET: Registered protocol family 17 [ 3.005438] mpls_gso: MPLS GSO support [ 3.011966] RAS: Correctable Errors collector initialized. [ 3.014862] AVX version of gcm_enc/dec engaged. [ 3.016443] AES CTR mode by8 optimization enabled [ 3.099436] sched_clock: Marking stable (3099409397, 0)->(3942451090, -843041693) [ 3.103407] registered taskstats version 1 [ 3.105568] Loading compiled-in X.509 certificates [ 3.107991] zswap: loaded using pool lzo/zbud [ 3.133594] Key type big_key registered [ 3.147401] Key type encrypted registered [ 3.149294] ima: No TPM chip found, activating TPM-bypass! [ 3.151545] ima: Allocated hash algorithm: sha1 [ 3.155316] ima: No architecture policies found [ 3.156834] evm: Initialising EVM extended attributes: [ 3.158924] evm: security.selinux [ 3.160200] evm: security.ima [ 3.161309] evm: security.capability [ 3.162417] evm: HMAC attrs: 0x1 [ 3.164597] rtc_cmos 00:05: setting system clock to 2026-09-11 08:24:49 UTC (1789115089) [ 3.170654] debug: unmapping init [mem 0xffffffff94203000-0xffffffff943fffff] [ 3.175049] debug: unmapping init [mem 0xffffffff92f82000-0xffffffff93258fff] [ 3.184153] Write protecting the kernel read-only data: 28672k [ 3.187928] debug: unmapping init [mem 0xffffffff91603000-0xffffffff917fffff] [ 3.190933] debug: unmapping init [mem 0xffffffff91f14000-0xffffffff91ffffff] [ 3.224335] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.233419] systemd[1]: Detected virtualization kvm. [ 3.235428] systemd[1]: Detected architecture x86-64. [ 3.237541] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.264841] systemd[1]: No hostname configured. [ 3.266889] systemd[1]: Set hostname to . [ 3.269134] random: systemd: uninitialized urandom read (16 bytes read) [ 3.272079] systemd[1]: Initializing machine ID from random generator. [ 3.313795] random: ln: uninitialized urandom read (6 bytes read) [ 3.413095] random: systemd: uninitialized urandom read (16 bytes read) [ 3.415715] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.420542] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.424679] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.085935] device-mapper: uevent: version 1.0.3 [ 4.087433] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 4.805887] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.847747] virtio_net virtio0 ens2: renamed from eth0 [ 4.914889] scsi host0: ata_piix [ 4.944294] scsi host1: ata_piix [ 4.945874] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.948577] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.800393] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.667512] random: crng init done [ 9.668802] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.240884] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.558503] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.840880] SELinux: Disabled at runtime. [ 11.906177] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.917932] systemd[1]: Detected virtualization kvm. [ 11.920249] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.438775] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.442699] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.451184] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.457260] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.460473] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.468681] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.477939] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. [ 12.524159] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.082524] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.473395] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.496856] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.574945] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.593037] EDAC sbridge: Ver: 1.1.2 [ 14.659501] Key type dns_resolver registered [ 14.980265] NFS: Registering the id_resolver key type [ 14.983242] Key type id_resolver registered [ 14.985444] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg633-client login: [ 44.869513] libcfs: loading out-of-tree module taints kernel. [ 44.972633] Key type ._llcrypt registered [ 44.984744] Key type .llcrypt registered [ 45.207157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.217987] alg: No test for adler32 (adler32-zlib) [ 46.270236] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 46.659115] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 48.295333] Key type lgssc registered [ 49.103550] Lustre: Echo OBD driver; http://www.lustre.org/ [ 104.592698] hrtimer: interrupt took 10689562 ns [ 168.181833] Lustre: Mounted lustre-client - version 2.17.57_142_g748fc9f [ 172.911874] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 188.039968] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing check_logdir /tmp/testlogs/ [ 193.719103] Lustre: DEBUG MARKER: oleg633-client.virtnet: executing yml_node [ 194.020684] Lustre: lustre-OST0000-osc-ffff9fc2d0475800: disconnect after 24s idle [ 198.216736] Lustre: DEBUG MARKER: Client: 2.17.57.142 [ 200.948781] Lustre: DEBUG MARKER: MDS: 2.17.57.142 [ 203.629226] Lustre: DEBUG MARKER: OSS: 2.17.57.142 [ 205.363492] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 11 04:28:10 EDT 2026 [ 221.333805] Lustre: Unmounted lustre-client [ 241.635722] Key type lgssc unregistered [ 241.886926] LNet: 8424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 241.895540] LNetError: 8424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 241.909530] LNet: Removed LNI 192.168.206.33@tcp [ 242.760169] Key type .llcrypt unregistered [ 242.762937] Key type ._llcrypt unregistered [ 268.695143] Key type ._llcrypt registered [ 268.698667] Key type .llcrypt registered [ 268.962216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 268.984676] alg: No test for adler32 (adler32-zlib) [ 269.910317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 270.002404] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 270.044710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 274.146219] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 274.290226] LNet: 9310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 274.300720] LNetError: 9310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 274.323426] LNet: Removed LNI 192.168.206.33@tcp [ 274.931392] Key type .llcrypt unregistered [ 274.937972] Key type ._llcrypt unregistered [ 280.661563] Key type ._llcrypt registered [ 280.665610] Key type .llcrypt registered [ 282.053675] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:29:26 (1789115366) [ 282.525066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 282.552828] alg: No test for adler32 (adler32-zlib) [ 283.465564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.564121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.590491] LNet: 9728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.618480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.682561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 283.710846] LNet: 9735:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 290.581000] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:29:35 (1789115375) [ 290.774629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.808971] LNet: 10312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.874526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.968407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 297.793207] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:29:42 (1789115382) [ 298.141926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.182758] LNet: 10897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 298.238483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.347027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 306.078605] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:29:50 (1789115390) [ 306.174555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.211624] LNet: 11474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.248613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.381505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 312.923989] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:29:58 (1789115398) [ 313.219692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.236574] LNet: 12066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.266160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.348385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 313.965180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 321.881430] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:30:06 (1789115406) [ 322.023433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.080262] LNet: 12653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.143705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.301620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 322.993781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 323.642638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 331.361506] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:30:16 (1789115416) [ 331.527433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.641778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.773961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 338.326045] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:30:23 (1789115423) [ 338.412526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.446693] LNet: 13832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.455512] LNet: 13832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 338.479521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.565717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 338.593969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 338.910821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 338.943465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 339.309447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 339.358462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 339.802437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 339.841271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 340.452396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 340.523493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 340.879517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 340.912902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 341.487823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 341.535226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 342.068594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 342.110236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 342.742470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 342.790882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 343.329630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 343.412164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 343.891544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 343.925786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 344.524419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 344.584718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 345.163431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 345.203467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 352.823296] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:30:37 (1789115437) [ 353.079638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.156138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.263715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 353.307352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 361.737758] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:30:46 (1789115446) [ 361.838357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.876664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.950758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 362.032092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 371.340473] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:30:55 (1789115455) [ 371.394217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.436034] LNet: 15690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.437802] LNet: 15690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 371.474910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.566384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 371.590892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 380.472671] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:31:05 (1789115465) [ 380.860351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.925695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.024392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 381.092558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 388.365624] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:31:13 (1789115473) [ 388.516169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.571517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.703283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 388.751637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 396.456081] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:31:21 (1789115481) [ 396.551903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.609489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.750982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 396.808361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 404.394777] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:31:28 (1789115488) [ 404.484495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.541683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.691926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 405.259775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 412.645916] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:31:37 (1789115497) [ 412.766527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.807314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.013760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 413.246368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 420.505283] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:31:45 (1789115505) [ 420.735894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.785222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.934863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 421.166890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 427.849423] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:31:52 (1789115512) [ 427.976746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.050707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.598517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.834763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.336777] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:32:01 (1789115521) [ 436.688064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.742393] LNet: 20527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 436.752523] LNet: 20527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 436.783460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.877055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.952811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.778112] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:32:09 (1789115529) [ 445.039148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.075717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.143298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.195801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.398772] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:32:17 (1789115537) [ 452.506284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.562255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.748281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 459.067817] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:32:23 (1789115543) [ 459.182368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.240976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.366086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 466.350859] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:32:31 (1789115551) [ 466.740933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.785833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.876355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 467.246481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 474.224601] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:32:39 (1789115559) [ 474.454283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.508256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.584618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 474.622121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 480.558435] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:32:45 (1789115565) [ 480.779305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.823756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.916151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 480.945216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 487.301098] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:32:52 (1789115572) [ 487.572796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.645728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.768712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 487.808267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 493.472848] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:32:58 (1789115578) [ 493.540780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.562093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.639218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 493.673686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 499.252632] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:33:04 (1789115584) [ 499.422916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.461128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.501755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 499.571831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 499.589307] LNetError: 25842:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 499.624972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 499.649717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 506.114816] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:33:11 (1789115591) [ 506.196646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.243473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.359851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YcgkahsB4y [ 506.416655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YcgkahsB4y [ 506.464560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YcgkahsB4y [ 513.578736] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:33:18 (1789115598) [ 513.645483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.714404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.860867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 513.941467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 513.998735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 520.620918] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 04:33:25 (1789115605) [ 520.752562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.859423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.013421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.038403] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 522.604212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 523.938252] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 525.919659] Key type lgssc registered [ 527.169801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 531.177295] LNetError: 27602:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.33@tcp99 has no available nets [ 533.917979] Key type lgssc unregistered [ 534.275210] LNetError: 28713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.289401] LNet: Removed LNI 192.168.206.33@tcp [ 534.919786] Key type .llcrypt unregistered [ 534.925903] Key type ._llcrypt unregistered [ 541.068931] Key type ._llcrypt registered [ 541.072694] Key type .llcrypt registered [ 542.279974] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:33:47 (1789115627) [ 542.806348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 542.818749] alg: No test for adler32 (adler32-zlib) [ 543.798948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.874524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 546.756215] LNet: Message forwarding will be enabled in 2s [ 549.855114] LNet: Message forwarding enabled [ 550.164776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 550.181536] LNet: Message forwarding will stop in 2s to 6s [ 554.602298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 555.638071] LNet: Message forwarding disabled [ 556.688926] LNet: Message forwarding will be enabled in 2s [ 559.775103] LNet: Message forwarding enabled [ 563.731287] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:34:08 (1789115648) [ 563.805655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.846190] LNet: 30114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.802022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.906633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 570.801400] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:34:16 (1789115656) [ 571.191987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.209668] LNet: 30704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.227104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.264048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 571.298535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 571.337404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 571.366947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 571.408882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 571.452166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 571.494900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 571.536990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 571.583984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 571.625046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 571.669501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 571.708592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 571.770712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 571.859549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 571.914401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 571.944538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 571.971945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 571.991262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 572.024019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 572.039908] LNetError: 30747:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 579.125831] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:34:23 (1789115663) [ 579.328622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.379290] LNet: 31318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 579.428613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.515349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 587.282594] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:34:32 (1789115672) [ 590.303582] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg633-server [ 592.454551] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:34:37 (1789115677) [ 592.745716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.781991] LNet: 32249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.815478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.867156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 592.897523] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 594.061150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[34-38]@tcp [ 594.089439] LNet: 32383:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 594.133211] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.34@tcp is being used as a gateway but routing feature is not turned on [ 594.164393] LNetError: 32253:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.34@tcp has gone from up to down [ 595.188839] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 595.217375] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 596.213668] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.36@tcp is being used as a gateway but routing feature is not turned on [ 596.229083] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 598.246224] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.38@tcp is being used as a gateway but routing feature is not turned on [ 598.254742] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 601.057898] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:34:46 (1789115686) [ 602.344157] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.35@tcp is being used as a gateway but routing feature is not turned on [ 602.362492] LNetError: 32253:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 603.339346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg633-server [ 605.315749] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:34:50 (1789115690) [ 605.391329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.416495] LNet: 33312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.426234] LNetError: 33312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.453778] LNet: Removed LNI 192.168.206.33@tcp [ 605.476192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.540910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 605.564255] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 606.081276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[34-39/2]@tcp [ 606.095772] LNet: 33436:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 606.117596] LNet: 33436:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 606.134194] LNetError: 33323:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.34@tcp has gone from up to down [ 606.145218] LNetError: 33323:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 606.182335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[34-39/2]@tcp [ 612.826857] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:34:57 (1789115697) [ 613.091795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.111177] LNetError: 34028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.128432] LNet: Removed LNI 192.168.206.33@tcp [ 613.147151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.250119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 613.301542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.346896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.412796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.474691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.494492] LNet: 34054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.499194] LNet: 34054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 613.514407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.575664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 613.697454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.738769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 613.814469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 613.877630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.929449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.004449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 614.122679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.184543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.273575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 614.357783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.409822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.452784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 614.544877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.588278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.675984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 614.772792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.806859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.847687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 614.922347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.947892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.019583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 615.067165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.103897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.166366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 615.211690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.241779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.292497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.071421] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:35:06 (1789115706) [ 623.357764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg633-server [ 625.077767] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:35:10 (1789115710) [ 626.911469] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg633-server [ 628.583234] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:35:13 (1789115713) [ 629.800084] Key type .llcrypt unregistered [ 629.803548] Key type ._llcrypt unregistered [ 636.632157] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 636.838213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 636.850865] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 636.857668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 639.416592] Key type ._llcrypt registered [ 639.418898] Key type .llcrypt registered [ 639.599582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 639.614277] alg: No test for adler32 (adler32-zlib) [ 640.504635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.584237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.606455] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 641.362041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 641.388649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 642.948740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 642.963231] LNet: Removed LNI 10.1.2.3@tcp [ 643.037298] LNet: 36153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.042352] LNetError: 36153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 643.055237] LNet: Removed LNI 192.168.206.33@tcp [ 643.444965] Key type .llcrypt unregistered [ 643.449035] Key type ._llcrypt unregistered [ 653.015408] Key type ._llcrypt registered [ 653.016675] Key type .llcrypt registered [ 654.204735] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:35:39 (1789115739) [ 654.653490] Key type .llcrypt unregistered [ 654.659131] Key type ._llcrypt unregistered [ 660.069539] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 660.188122] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 660.207342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 660.211484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 662.828071] Key type ._llcrypt registered [ 662.829863] Key type .llcrypt registered [ 662.985867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.004953] alg: No test for adler32 (adler32-zlib) [ 663.924969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.015310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.053083] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 664.740152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 664.759472] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 667.258923] LNet: 37656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.266361] LNetError: 37656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 667.271971] LNet: Removed LNI 10.1.2.3@tcp [ 667.284874] LNet: Removed LNI 192.168.206.33@tcp [ 667.985477] Key type .llcrypt unregistered [ 667.987566] Key type ._llcrypt unregistered [ 679.375287] Key type ._llcrypt registered [ 679.381185] Key type .llcrypt registered [ 680.698686] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:36:05 (1789115765) [ 681.331229] Key type .llcrypt unregistered [ 681.333950] Key type ._llcrypt unregistered [ 686.960417] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.070785] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.077467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.084398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.647172] Key type ._llcrypt registered [ 689.651501] Key type .llcrypt registered [ 689.804458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.824044] alg: No test for adler32 (adler32-zlib) [ 690.716401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.764309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 690.783868] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 692.331155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.346764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.021145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 694.064307] LNet: Removed LNI 10.1.2.3@tcp [ 694.093760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 694.150565] LNet: Removed LNI 10.1.2.31@tcp [ 694.168840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 694.200641] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.803880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 694.822462] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 695.699101] LNet: 39581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.705872] LNetError: 39581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 695.710311] LNet: Removed LNI 10.1.2.31@tcp [ 695.720767] LNet: Removed LNI 10.1.2.3@tcp [ 696.259105] Key type .llcrypt unregistered [ 696.261399] Key type ._llcrypt unregistered [ 707.687435] Key type ._llcrypt registered [ 707.690429] Key type .llcrypt registered [ 708.989370] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:36:33 (1789115793) [ 709.517989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 709.555826] alg: No test for adler32 (adler32-zlib) [ 710.518664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.567885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.603428] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 711.473863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.498567] LNet: 40538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.509420] LNetError: 40538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.525805] LNet: Removed LNI 192.168.206.33@tcp [ 711.555040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.617479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.33@tcp [ 711.646795] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 712.559314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.33@tcp [ 712.590363] LNet: Removed LNI 192.168.206.33@tcp [ 712.655336] LNet: 40673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.311460] Key type .llcrypt unregistered [ 713.318317] Key type ._llcrypt unregistered [ 724.591598] Key type ._llcrypt registered [ 724.593644] Key type .llcrypt registered [ 725.542931] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:36:50 (1789115810) [ 727.536561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg633-server [ 729.053880] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:36:54 (1789115814) [ 729.733300] Key type .llcrypt unregistered [ 729.735630] Key type ._llcrypt unregistered [ 735.895521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 736.179104] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 736.191954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 736.200862] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 738.905619] Key type ._llcrypt registered [ 738.908196] Key type .llcrypt registered [ 739.091643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.109674] alg: No test for adler32 (adler32-zlib) [ 740.014334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.057919] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 740.812172] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 743.147351] LNet: 42518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.153440] LNetError: 42518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.166416] LNet: Removed LNI 192.168.206.33@tcp1 [ 743.179735] LNet: Removed LNI 10.1.2.3@tcp2 [ 743.878395] Key type .llcrypt unregistered [ 743.882481] Key type ._llcrypt unregistered [ 755.932734] Key type ._llcrypt registered [ 755.934248] Key type .llcrypt registered [ 757.160157] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:37:22 (1789115842) [ 757.494753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 757.512639] alg: No test for adler32 (adler32-zlib) [ 758.413993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.495919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.530851] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 759.383686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.402699] LNet: 43492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.414161] LNetError: 43492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.437323] LNet: Removed LNI 192.168.206.33@tcp [ 759.458129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.532644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 759.565926] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 765.828074] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:37:30 (1789115850) [ 765.949639] LNet: 44077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.956260] LNetError: 44077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 765.975579] LNet: Removed LNI 192.168.206.33@tcp [ 766.852393] Key type .llcrypt unregistered [ 766.856958] Key type ._llcrypt unregistered [ 772.750317] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 772.980793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.995781] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 773.015254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 775.533705] Key type ._llcrypt registered [ 775.537354] Key type .llcrypt registered [ 775.715258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.732336] alg: No test for adler32 (adler32-zlib) [ 776.648842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.694514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.714522] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 777.403516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 777.419697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 779.473448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.502851] LNet: 44830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.514717] LNetError: 44830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 779.531757] LNet: Removed LNI 10.1.2.3@tcp [ 779.546833] LNet: Removed LNI 192.168.206.33@tcp [ 779.582586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.666911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 779.691071] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 779.699556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 780.299692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.325393] LNet: 44857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.331572] LNetError: 44857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 780.336813] LNetError: 44857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 780.345495] LNet: Removed LNI 10.1.2.3@tcp [ 780.354846] LNet: Removed LNI 192.168.206.33@tcp [ 780.381352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.476154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 780.508443] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 780.521787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 781.152231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 781.180832] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 782.087616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.110569] LNet: 45019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.117939] LNetError: 45019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 782.122577] LNetError: 45019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 782.128877] LNet: Removed LNI 10.1.2.3@tcp [ 782.131881] LNet: Removed LNI 192.168.206.33@tcp [ 782.140060] LNet: Removed LNI 192.168.206.33@tcp2 [ 782.168942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.254360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 782.278489] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 782.288435] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.293581] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 789.443125] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:37:54 (1789115874) [ 789.846797] LNet: 45617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.855250] LNetError: 45617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 789.865872] LNetError: 45617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 789.873483] LNet: Removed LNI 10.1.2.3@tcp [ 789.875577] LNet: Removed LNI 192.168.206.33@tcp [ 789.889586] LNet: Removed LNI 192.168.206.33@tcp2 [ 790.455168] Key type .llcrypt unregistered [ 790.459591] Key type ._llcrypt unregistered [ 796.322486] Key type ._llcrypt registered [ 796.325524] Key type .llcrypt registered [ 796.472717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.481385] alg: No test for adler32 (adler32-zlib) [ 797.436473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.495154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.508049] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 798.154544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.170884] LNet: 46016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.178383] LNetError: 46016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.196848] LNet: Removed LNI 192.168.206.33@tcp [ 798.240553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.305712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 798.335599] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 799.082666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.100387] LNet: 46046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.106900] LNetError: 46046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.125463] LNet: Removed LNI 192.168.206.33@tcp [ 799.141299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.215476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 799.233744] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 799.715609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.752679] LNet: Removed LNI 192.168.206.33@tcp [ 799.772849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.833128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 799.853836] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 800.435026] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 800.580344] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 800.592062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 800.626829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 802.924822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.992421] LNet: 46156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.995902] LNet: 46156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 803.000591] LNetError: 46156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.004213] LNetError: 46156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 803.019536] LNet: Removed LNI 192.168.206.33@tcp [ 803.053861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.143286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 803.205248] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 812.267336] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:38:16 (1789115896) [ 812.678665] LNet: 46753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 812.695457] LNetError: 46753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 812.730279] LNet: Removed LNI 192.168.206.33@tcp [ 813.792160] Key type .llcrypt unregistered [ 813.793787] Key type ._llcrypt unregistered [ 822.312226] Key type ._llcrypt registered [ 822.316333] Key type .llcrypt registered [ 822.469884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.478681] alg: No test for adler32 (adler32-zlib) [ 823.400475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.454689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 823.475918] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 824.242775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.268433] LNet: 47157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.272628] LNetError: 47157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.283764] LNet: Removed LNI 192.168.206.33@tcp [ 824.310824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.435737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 824.502748] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 832.619871] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:38:37 (1789115917) [ 833.024444] LNet: 47750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.033831] LNetError: 47750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.050196] LNet: Removed LNI 192.168.206.33@tcp [ 833.769345] Key type .llcrypt unregistered [ 833.774360] Key type ._llcrypt unregistered [ 839.949589] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.048026] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.050756] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.064233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.717228] Key type ._llcrypt registered [ 842.720137] Key type .llcrypt registered [ 842.948124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.976313] alg: No test for adler32 (adler32-zlib) [ 843.927387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.968160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.998898] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 844.942371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.985512] LNet: 48202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.998869] LNetError: 48202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.012231] LNet: Removed LNI 192.168.206.33@tcp [ 845.059504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.144089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 845.167308] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 852.447595] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:38:57 (1789115937) [ 852.665491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.680909] LNet: 48801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.685680] LNetError: 48801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.701590] LNet: Removed LNI 192.168.206.33@tcp [ 852.738142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.838357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 859.867569] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:39:04 (1789115944) [ 859.985152] LNet: 49377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.929136] Key type .llcrypt unregistered [ 860.935076] Key type ._llcrypt unregistered [ 868.135815] Key type ._llcrypt registered [ 868.138172] Key type .llcrypt registered [ 868.358187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.385898] alg: No test for adler32 (adler32-zlib) [ 869.294455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.382518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.404096] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 870.421258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.441470] LNet: 49785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.446994] LNetError: 49785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.466531] LNet: Removed LNI 192.168.206.33@tcp [ 870.489442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.574126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 870.605958] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 877.424561] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:39:22 (1789115962) [ 877.584865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.615315] LNet: 50374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.627052] LNetError: 50374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.652828] LNet: Removed LNI 192.168.206.33@tcp [ 877.681743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.777892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 884.857991] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:39:29 (1789115969) [ 886.494463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 888.303705] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:39:33 (1789115973) [ 889.979125] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 891.765474] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:39:36 (1789115976) [ 893.711541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 895.452290] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:39:40 (1789115980) [ 896.890764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 898.656781] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:39:43 (1789115983) [ 900.302709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 902.198937] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:39:47 (1789115987) [ 902.678494] LNet: 52432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.362713] Key type .llcrypt unregistered [ 903.371062] Key type ._llcrypt unregistered [ 909.312775] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.394883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.407275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.412235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.116635] Key type ._llcrypt registered [ 912.117927] Key type .llcrypt registered [ 912.323321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.360259] alg: No test for adler32 (adler32-zlib) [ 913.353978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.472379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.506741] LNet: Added LNI 192.168.206.33@tcp [8/512/0/180] [ 913.510732] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 913.518344] LNet: 52754:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 913.648989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.677302] LNet: 52773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.690857] LNetError: 52773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 913.705696] LNet: Removed LNI 10.1.2.3@tcp [ 913.710373] LNetError: 52773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.729148] LNet: Removed LNI 192.168.206.33@tcp [ 913.764339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.855831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 913.891486] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 914.016366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.050130] LNet: 52797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.065304] LNetError: 52797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.085185] LNet: Removed LNI 192.168.206.33@tcp [ 914.105884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.219351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.258451] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 914.262932] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 914.266940] LNet: 52805:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 914.366972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.389945] LNet: 52823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.398656] LNetError: 52823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.419093] LNet: Removed LNI 10.1.2.3@tcp [ 914.422071] LNetError: 52823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.440956] LNet: Removed LNI 192.168.206.33@tcp [ 914.492871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.600525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 914.627678] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 914.633593] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 914.637403] LNet: 52831:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 914.834204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.853393] LNet: 52857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.858943] LNetError: 52857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 914.866198] LNet: Removed LNI 10.1.2.3@tcp [ 914.869141] LNetError: 52857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.898692] LNet: Removed LNI 192.168.206.33@tcp [ 914.926557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.999437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 915.025563] LNet: Added LNI 192.168.206.33@tcp [8/256/0/182] [ 915.029913] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 915.122657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.143978] LNet: 52885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.151439] LNetError: 52885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 915.155850] LNet: Removed LNI 10.1.2.3@tcp [ 915.159407] LNetError: 52885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.173229] LNet: Removed LNI 192.168.206.33@tcp [ 915.208386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.318694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 915.365735] LNet: Added LNI 192.168.206.33@tcp [8/256/0/182] [ 915.373351] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 915.379518] LNet: 52893:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 915.503725] LNet: Removed LNI 10.1.2.3@tcp [ 915.519322] LNet: Removed LNI 192.168.206.33@tcp [ 916.259741] Key type .llcrypt unregistered [ 916.265551] Key type ._llcrypt unregistered [ 927.792573] Key type ._llcrypt registered [ 927.795331] Key type .llcrypt registered [ 928.897715] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 04:40:14 (1789116014) [ 930.302451] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 932.066261] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:40:17 (1789116017) [ 932.741801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 932.770827] alg: No test for adler32 (adler32-zlib) [ 933.741688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.785864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.808466] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 934.468427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.33 [ 934.506426] LNet: 54162:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 934.519160] LNet: Removed LNI 192.168.206.33@tcp [ 941.029356] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:40:25 (1789116025) [ 941.140240] LNet: 54723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.155111] LNetError: 54723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.191658] LNet: Removed LNI 192.168.206.33@tcp [ 941.870338] Key type .llcrypt unregistered [ 941.874938] Key type ._llcrypt unregistered [ 949.607362] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.883214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 952.622782] Key type ._llcrypt registered [ 952.628806] Key type .llcrypt registered [ 952.803994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.823533] alg: No test for adler32 (adler32-zlib) [ 954.000761] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.409852] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:40:46 (1789116046) [ 961.887804] LNet: 55649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.898527] LNetError: 55649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.925307] LNet: Removed LNI 10.1.2.3@tcp [ 962.653461] Key type .llcrypt unregistered [ 962.656915] Key type ._llcrypt unregistered [ 969.609582] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 969.884793] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 969.890597] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 969.894810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 972.787148] Key type ._llcrypt registered [ 972.790698] Key type .llcrypt registered [ 972.997657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.014636] alg: No test for adler32 (adler32-zlib) [ 974.131784] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 980.901529] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:41:06 (1789116066) [ 982.788749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 984.633772] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:41:09 (1789116069) [ 984.856877] LNet: 56839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.863878] LNetError: 56839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 984.882228] LNet: Removed LNI 10.1.2.3@tcp [ 985.449820] Key type .llcrypt unregistered [ 985.455300] Key type ._llcrypt unregistered [ 990.700873] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 990.899129] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 990.912686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 990.919819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.729097] Key type ._llcrypt registered [ 993.732179] Key type .llcrypt registered [ 993.907435] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.920952] alg: No test for adler32 (adler32-zlib) [ 994.848493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.915883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.296105] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:41:28 (1789116088) [ 1004.231680] LNet: 58072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1004.245399] LNetError: 58072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1004.267929] LNet: Removed LNI 10.1.2.3@tcp [ 1005.247318] Key type .llcrypt unregistered [ 1005.251789] Key type ._llcrypt unregistered [ 1011.610749] Key type ._llcrypt registered [ 1011.616327] Key type .llcrypt registered [ 1011.841769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1011.850341] alg: No test for adler32 (adler32-zlib) [ 1012.838411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1012.879963] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1020.765620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1021.855317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_interrupt -r 1 [ 1021.922835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_interrupt -r 1 [ 1021.958314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1021.973919] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1022.112254] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1022.209298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1022.240324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1022.271818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1022.302864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1022.346219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1022.378247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1022.423324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1022.461183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1022.475810] LNet: 58514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1022.489408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1022.522924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1022.568823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1022.616337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1022.685602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.726829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1022.761703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1023.299762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_dropped -r 1 [ 1023.334375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_dropped -r 1 [ 1023.382772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1023.415229] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1023.425783] LNet: Skipped 1 previous similar message [ 1023.714292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.758259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1023.789577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.816592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.855277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.890686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.926399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.959144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.990910] LNet: 58623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.014917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.060792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.113766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.150965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.193969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.241027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1024.266960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1024.676599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_aborted -r 1 [ 1024.706640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_aborted -r 1 [ 1024.745259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1024.757144] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1024.931633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.956621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1024.981509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.004543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.027678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.058786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.078682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.097045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.109755] LNet: 58732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.132251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.157247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.192258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.227148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.262704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.287830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.338633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1025.884397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_no_route -r 1 [ 1025.950947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_no_route -r 1 [ 1025.996842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1026.294851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.332181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.383854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.418081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.475944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.508646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.537949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.569154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.600126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.633659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.670758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.720636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.768653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.798852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.829771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1027.443850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_timeout -r 1 [ 1027.484289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_timeout -r 1 [ 1027.528649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1027.549269] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1027.556673] LNet: Skipped 2 previous similar messages [ 1027.778593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.816964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1027.859347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.915361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1027.974252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.015172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.056723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.104273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.125438] LNet: 58951:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.133946] LNet: 58951:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1028.162518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.204069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.249803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.277959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.318107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.352984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.381832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1028.836693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_error -r 1 [ 1028.884326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_error -r 1 [ 1028.923904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1029.153687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.180193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1029.202759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1029.226762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.258823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1029.286704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.313596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.337727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.369172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.402443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.424875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.451862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.475716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.497295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.515281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1029.772657] LNet: 59105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.779784] LNetError: 59105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.801834] LNet: Removed LNI 192.168.206.33@tcp [ 1030.188355] Key type .llcrypt unregistered [ 1030.193766] Key type ._llcrypt unregistered [ 1050.524936] Key type ._llcrypt registered [ 1050.528989] Key type .llcrypt registered [ 1052.107581] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:42:16 (1789116136) [ 1053.146478] Key type .llcrypt unregistered [ 1053.149464] Key type ._llcrypt unregistered [ 1059.190908] Key type ._llcrypt registered [ 1059.195290] Key type .llcrypt registered [ 1059.504931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1059.526973] alg: No test for adler32 (adler32-zlib) [ 1060.416650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1060.446648] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1067.702653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1078.444668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.523515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1078.549109] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1079.945422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_interrupt -r 1 [ 1079.996686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 1 [ 1080.027326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_interrupt -r 1 [ 1080.065332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 1 [ 1080.122449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_interrupt -r 1 [ 1080.160962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 1 [ 1080.207309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_interrupt -r 1 [ 1080.249729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 1 [ 1080.285195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1080.307758] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1080.316384] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1080.538450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1080.572343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1080.610541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1080.638484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1080.687903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1080.734048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1080.763664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1080.811877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1080.855345] LNet: 60579:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1080.897556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1080.948453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1080.990848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.032808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.087244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.132943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.195449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.820325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_dropped -r 1 [ 1081.851831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 1 [ 1081.914020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_dropped -r 1 [ 1081.966246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 1 [ 1082.012749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_dropped -r 1 [ 1082.068625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 1 [ 1082.121201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_dropped -r 1 [ 1082.164236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 1 [ 1082.209269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1082.230609] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1082.238870] LNet: Skipped 2 previous similar messages [ 1082.468306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.504711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.534406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.568427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.606533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.642182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1082.677274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1082.699276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1082.724781] LNet: 60700:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1082.748174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1082.782469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1082.823657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1082.868182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1082.913104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1082.942943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1082.966972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1083.513314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_aborted -r 1 [ 1083.549418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 1 [ 1083.591100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_aborted -r 1 [ 1083.641945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 1 [ 1083.685223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_aborted -r 1 [ 1083.710771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 1 [ 1083.778607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_aborted -r 1 [ 1083.834743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 1 [ 1083.870543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1083.890506] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1083.892788] LNet: Skipped 2 previous similar messages [ 1084.079727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.106487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.137289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.166400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.200607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.249574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.309892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.353862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.373880] LNet: 60822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.395412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.427384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.469992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.524532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.558951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.602593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.637478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1085.192760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_no_route -r 1 [ 1085.253245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 1 [ 1085.293939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_no_route -r 1 [ 1085.341444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 1 [ 1085.379972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_no_route -r 1 [ 1085.442759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 1 [ 1085.485774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_no_route -r 1 [ 1085.538687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 1 [ 1085.576731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1085.805293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.834610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.858540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.884706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.914864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.946289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.971102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1085.995971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.025634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.059954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.095712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.125673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.169947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.201458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.238923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1086.772756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_timeout -r 1 [ 1086.811192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 1 [ 1086.864706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_timeout -r 1 [ 1086.907606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 1 [ 1086.947200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_timeout -r 1 [ 1086.980803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 1 [ 1087.008405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_timeout -r 1 [ 1087.033521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 1 [ 1087.067451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1087.091424] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1087.095958] LNet: Skipped 5 previous similar messages [ 1087.306748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.343583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.375065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.402513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.427645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.454603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.497496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.536914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.559053] LNet: 61064:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.563732] LNet: 61064:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1087.584672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.617291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.670995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.708214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.740767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.787346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1087.821830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1088.311471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_error -r 1 [ 1088.350737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_error -r 1 [ 1088.394451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_error -r 1 [ 1088.438208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_error -r 1 [ 1088.476320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_error -r 1 [ 1088.511709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_error -r 1 [ 1088.560939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_error -r 1 [ 1088.600051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_error -r 1 [ 1088.648151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1088.878988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1088.910645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1088.964471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.009153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.054264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.099389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.140622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.163968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.196249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.233991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.270572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.301106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.339481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.393405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.453628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1095.082368] LNet: 61266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.089719] LNetError: 61266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.097489] LNet: Removed LNI 192.168.206.33@tcp [ 1096.165731] LNet: Removed LNI 192.168.206.33@tcp1 [ 1096.561204] Key type .llcrypt unregistered [ 1096.563745] Key type ._llcrypt unregistered [ 1111.935153] Key type ._llcrypt registered [ 1111.937443] Key type .llcrypt registered [ 1112.963755] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:43:18 (1789116198) [ 1113.915439] Key type .llcrypt unregistered [ 1113.917829] Key type ._llcrypt unregistered [ 1120.356214] Key type ._llcrypt registered [ 1120.358622] Key type .llcrypt registered [ 1120.648154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.663085] alg: No test for adler32 (adler32-zlib) [ 1121.563790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1121.583741] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1129.131161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1129.854982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_dropped -r 1 [ 1129.892713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_dropped -r 1 [ 1129.953799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1129.986954] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1130.199949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1130.232278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.263174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.306705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.346289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.383713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.424730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.452405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.466342] LNet: 62521:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1130.478889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.502985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.531265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.564469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.592676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.634207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.670737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1131.148421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_error -r 1 [ 1131.203231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_error -r 1 [ 1131.253562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1131.277947] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1131.553547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1131.592758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1131.630779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1131.670823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1131.718292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1131.749563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1131.777552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1131.827337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1131.856294] LNet: 62630:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1131.888409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1131.929423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1131.972819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.010411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.053604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.100449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.145423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1132.614790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_timeout -r 1 [ 1132.647419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_timeout -r 1 [ 1132.677248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1132.693513] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1132.890540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.915455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.938286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.965371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.992593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.035662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.063755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.101102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.127321] LNet: 62739:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.151861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.189888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.230250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.284508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.327763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.357491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.384741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1133.633212] LNet: 62784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.641428] LNetError: 62784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.664112] LNet: Removed LNI 192.168.206.33@tcp [ 1134.029556] Key type .llcrypt unregistered [ 1134.032412] Key type ._llcrypt unregistered [ 1150.193281] Key type ._llcrypt registered [ 1150.196863] Key type .llcrypt registered [ 1152.052644] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:43:56 (1789116236) [ 1153.984430] Key type .llcrypt unregistered [ 1153.995569] Key type ._llcrypt unregistered [ 1162.100815] Key type ._llcrypt registered [ 1162.105291] Key type .llcrypt registered [ 1162.410858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.428919] alg: No test for adler32 (adler32-zlib) [ 1163.545739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1163.584743] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1173.207915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1183.906972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1183.958174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1184.009285] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1185.703794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_dropped -r 1 [ 1185.747906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 1 [ 1185.784476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_dropped -r 1 [ 1185.833325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 1 [ 1185.912560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_dropped -r 1 [ 1185.963557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 1 [ 1186.010444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_dropped -r 1 [ 1186.060390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 1 [ 1186.124561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1186.155312] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1186.165155] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 1186.388937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.444254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.490700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.520834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.577710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.612051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.658638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.722635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.733826] LNet: 64264:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.743990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.766214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.791133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.821975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.850511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.875987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.901330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.924918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.480271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_error -r 1 [ 1187.514931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_error -r 1 [ 1187.555857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_error -r 1 [ 1187.608944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_error -r 1 [ 1187.653499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_error -r 1 [ 1187.718398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_error -r 1 [ 1187.770961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_error -r 1 [ 1187.825439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_error -r 1 [ 1187.919473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1187.953916] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1187.966362] LNet: Skipped 2 previous similar messages [ 1188.208945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.238927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.286367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.351139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.387586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.410313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.434600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.460888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.477624] LNet: 64387:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.497311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.546305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.584256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.632729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.696281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.757028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.805938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1188.879702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.522217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_timeout -r 1 [ 1189.570250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_timeout -r 1 [ 1189.621812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_timeout -r 1 [ 1189.656306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_timeout -r 1 [ 1189.706333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_timeout -r 1 [ 1189.757303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_timeout -r 1 [ 1189.794809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_timeout -r 1 [ 1189.849307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_timeout -r 1 [ 1189.889254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.133@tcp [ 1189.904559] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1189.907858] LNet: Skipped 1 previous similar message [ 1190.110507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.144728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.200399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.241702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.296428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.359845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.415766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.453308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.474474] LNet: 64510:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.505582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.549315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.616724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.654308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.687677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.728948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.766709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1190.809605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.440369] LNet: 64592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1196.449921] LNetError: 64592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.458738] LNet: Removed LNI 192.168.206.33@tcp [ 1196.473453] LNet: Removed LNI 192.168.206.33@tcp1 [ 1196.864607] Key type .llcrypt unregistered [ 1196.867556] Key type ._llcrypt unregistered [ 1213.737261] Key type ._llcrypt registered [ 1213.739144] Key type .llcrypt registered [ 1215.070198] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:44:59 (1789116299) [ 1216.713662] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1218.936386] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:45:03 (1789116303) [ 1219.874647] Key type .llcrypt unregistered [ 1219.880681] Key type ._llcrypt unregistered [ 1226.520318] Key type ._llcrypt registered [ 1226.523691] Key type .llcrypt registered [ 1226.798875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1226.812451] alg: No test for adler32 (adler32-zlib) [ 1227.746322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1227.803336] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1235.234195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1235.943352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e network_timeout -r 1 [ 1236.012093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e network_timeout -r 1 [ 1236.089081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1236.129115] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1236.353677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.383423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.408797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.432038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.467410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.515532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.552667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.589685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.614715] LNet: 66141:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.657951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.712692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.746835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.771700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.798137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.854064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.176152] LNet: 66184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.189221] LNetError: 66184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.216107] LNet: Removed LNI 192.168.206.33@tcp [ 1237.608517] Key type .llcrypt unregistered [ 1237.610803] Key type ._llcrypt unregistered [ 1257.012728] Key type ._llcrypt registered [ 1257.016738] Key type .llcrypt registered [ 1257.342304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1257.361562] alg: No test for adler32 (adler32-zlib) [ 1258.308545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1258.337705] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1266.139974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1277.025069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.099320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1277.139814] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1279.099619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e network_timeout -r 1 [ 1279.167065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e network_timeout -r 1 [ 1279.248672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e network_timeout -r 1 [ 1279.316838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e network_timeout -r 1 [ 1279.381583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e network_timeout -r 1 [ 1279.439576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e network_timeout -r 1 [ 1279.502377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e network_timeout -r 1 [ 1279.552265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e network_timeout -r 1 [ 1279.597236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1279.660060] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1279.840456] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1279.846464] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 1279.974197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.007787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1280.050241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.081451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.114836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.152092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.189368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.226278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.251395] LNet: 67069:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.281800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.330924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.396981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.468073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.556683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.613381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.169247] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1286.666199] LNet: 67147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1286.674242] LNetError: 67147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1286.685259] LNet: Removed LNI 192.168.206.33@tcp [ 1286.697258] LNet: Removed LNI 192.168.206.33@tcp1 [ 1287.473578] Key type .llcrypt unregistered [ 1287.481403] Key type ._llcrypt unregistered [ 1306.479761] Key type ._llcrypt registered [ 1306.485247] Key type .llcrypt registered [ 1308.022366] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:46:32 (1789116392) [ 1308.619945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1308.655540] alg: No test for adler32 (adler32-zlib) [ 1309.511666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1309.625151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1309.673811] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1310.883225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1310.911023] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1312.290482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1312.420305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1312.553402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1312.624583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1312.669622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1312.729760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1312.779638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.33@tcp [ 1312.835994] LNetError: 68249:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1313.695384] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1313.696230] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1315.745049] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1316.583811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1319.840987] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1320.223240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1320.269737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1320.363375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1320.394456] LNet: 68357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1320.407263] LNetError: 68357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.439229] LNet: Removed LNI 192.168.206.33@tcp [ 1321.444610] LNetError: 68357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.468504] LNet: Removed LNI 192.168.206.33@tcp1 [ 1321.530656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.644512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1321.677236] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1322.604758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1322.627287] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1323.736796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1323.805667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1323.899808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1323.960584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1323.993215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1324.041707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1324.086145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1324.137399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.33@tcp [ 1324.157886] LNetError: 68630:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1324.639361] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1327.854988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1330.784423] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1330.791651] LNet: Skipped 1 previous similar message [ 1335.989753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1336.009615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.066953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.022529] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:47:10 (1789116430) [ 1346.238827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.294062] LNet: 69345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.309417] LNetError: 69345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.329027] LNet: Removed LNI 192.168.206.33@tcp [ 1346.362099] LNet: Removed LNI 192.168.206.33@tcp1 [ 1346.432041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.528924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1346.585902] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1347.811638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1347.859403] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1349.235334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1349.360295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1349.428224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1349.470518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1349.509646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1349.545386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1349.613764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.33@tcp --health 0 [ 1349.770424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.559282] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.33@tcp [ 1350.561308] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1350.582663] LNet: Skipped 1 previous similar message [ 1365.969833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1366.162807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1366.227747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1366.270823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.33@tcp --health 500 [ 1366.416861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1373.211889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1373.269150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.302961] LNet: 69841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.311922] LNetError: 69841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.317955] LNetError: 69841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1373.336152] LNet: Removed LNI 192.168.206.33@tcp [ 1373.355651] LNet: Removed LNI 192.168.206.33@tcp1 [ 1373.379764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.419837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.446382] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1374.225912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1374.254031] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1375.672763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1375.764657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.885509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1375.951811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1376.046965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1376.120775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1376.150991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.173209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.33@tcp --health 0 [ 1377.503379] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1377.516064] LNet: Skipped 3 previous similar messages [ 1378.723665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.215213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.254554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1388.334231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1396.758533] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:48:00 (1789116480) [ 1397.402964] LNet: 70887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.424409] LNetError: 70887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.432919] LNetError: 70887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1397.444464] LNet: Removed LNI 192.168.206.33@tcp [ 1397.484959] LNet: Removed LNI 192.168.206.33@tcp1 [ 1398.385665] Key type .llcrypt unregistered [ 1398.397437] Key type ._llcrypt unregistered [ 1405.525479] Key type ._llcrypt registered [ 1405.529279] Key type .llcrypt registered [ 1405.857485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1405.890941] alg: No test for adler32 (adler32-zlib) [ 1406.901766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1406.941863] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1416.993412] Lustre: DEBUG MARKER: Initial discovery [ 1417.036448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.133@tcp [ 1419.764221] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1419.835604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1419.921647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.133@tcp [ 1419.986255] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1420.024979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1420.101838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.133@tcp -r 1 -m GET -l 3 [ 1420.174481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.133@tcp -r 1 -m GET -e local_error [ 1420.255521] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1420.258958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.133@tcp -r 1 -m PUT -l 6 [ 1421.727300] Lustre: DEBUG MARKER: Force oleg633-client.virtnet to discover 192.168.206.133@tcp (in background) [ 1422.079206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.133@tcp [ 1423.461402] Lustre: DEBUG MARKER: Force oleg633-server to discover 192.168.206.33@tcp [ 1425.132508] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1426.059735] Lustre: DEBUG MARKER: Wait for 71656 [ 1427.246943] Lustre: DEBUG MARKER: Finished wait on 71656 [ 1427.527656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1427.545933] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1427.549211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1427.916419] LNet: 72108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.924732] LNetError: 72108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.947953] LNet: Removed LNI 192.168.206.33@tcp [ 1428.332805] Key type .llcrypt unregistered [ 1428.337878] Key type ._llcrypt unregistered [ 1447.511952] Key type ._llcrypt registered [ 1447.519231] Key type .llcrypt registered [ 1449.148953] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:48:53 (1789116533) [ 1449.675263] Key type .llcrypt unregistered [ 1449.679279] Key type ._llcrypt unregistered [ 1455.736140] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1455.874592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1455.882466] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1455.890070] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1458.542317] Key type ._llcrypt registered [ 1458.545985] Key type .llcrypt registered [ 1458.805451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1458.834836] alg: No test for adler32 (adler32-zlib) [ 1459.756280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.808314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.827110] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1460.400304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1460.413576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1468.716248] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:49:13 (1789116553) [ 1468.906718] LNet: 74192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.916180] LNetError: 74192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1468.927177] LNet: Removed LNI 10.1.2.3@tcp [ 1468.947822] LNet: Removed LNI 192.168.206.33@tcp [ 1469.629469] Key type .llcrypt unregistered [ 1469.636680] Key type ._llcrypt unregistered [ 1477.122925] Key type ._llcrypt registered [ 1477.124459] Key type .llcrypt registered [ 1477.344233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1477.364336] alg: No test for adler32 (adler32-zlib) [ 1478.308712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.357311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1478.374124] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1479.105672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1479.141735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1480.403712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.33@tcp [ 1481.503158] LNetError: Unexpected error -22 connecting to 192.168.206.33@tcp at host 192.168.206.33:988 [ 1487.406110] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:49:32 (1789116572) [ 1487.665247] LNet: 75329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.683665] LNetError: 75329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1487.698400] LNet: Removed LNI 10.1.2.3@tcp [ 1487.721708] LNet: Removed LNI 192.168.206.33@tcp [ 1488.540609] Key type .llcrypt unregistered [ 1488.545835] Key type ._llcrypt unregistered [ 1494.520401] Key type ._llcrypt registered [ 1494.528259] Key type .llcrypt registered [ 1494.773864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1494.793438] alg: No test for adler32 (adler32-zlib) [ 1495.748817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.830820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1495.869806] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1496.849801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1496.892024] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 1497.757097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.33@tcp1 --nid 192.168.206.33@tcp2 [ 1501.854989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp1 192.168.206.33@tcp2 [ 1501.902474] LNetError: 76169:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.33@tcp1 to 192.168.206.33@tcp2 - Route table may be misconfigured [ 1501.935832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp2 192.168.206.33@tcp1 [ 1508.901976] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:49:53 (1789116593) [ 1509.091418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.122223] LNet: 76737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.136631] LNetError: 76737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1509.155052] LNet: Removed LNI 192.168.206.33@tcp1 [ 1510.197204] LNet: Removed LNI 192.168.206.33@tcp2 [ 1510.240241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.354095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.408812] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1511.259211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1511.293808] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1512.234390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 1512.277585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.33@tcp [ 1512.311954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp1 192.168.206.33@tcp1 [ 1512.357350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.33@tcp -d 192.168.206.33@tcp -e network_timeout [ 1512.391589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -e network_timeout [ 1512.439434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -e network_timeout [ 1512.472795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -e network_timeout [ 1512.531543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1512.562753] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1512.585683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1512.696914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.801273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1513.380904] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1519.508147] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:50:04 (1789116604) [ 1519.595391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.622091] LNet: 77596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.631326] LNetError: 77596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.641722] LNetError: 77596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1519.659585] LNet: Removed LNI 192.168.206.33@tcp [ 1520.680408] LNet: Removed LNI 192.168.206.33@tcp1 [ 1520.733333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.941255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1521.342772] LNet: 77647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1521.861709] Key type .llcrypt unregistered [ 1521.872632] Key type ._llcrypt unregistered [ 1534.574789] Key type ._llcrypt registered [ 1534.576835] Key type .llcrypt registered [ 1535.861843] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:50:20 (1789116620) [ 1536.447214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1536.472062] alg: No test for adler32 (adler32-zlib) [ 1537.406792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.736981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1541.109397] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:50:25 (1789116625) [ 1541.317931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1541.335746] LNet: 78745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.353425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.407520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.427593] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1542.177547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1542.197705] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1542.963262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1543.032430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp1 [ 1543.097941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp1 [ 1551.348800] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:50:35 (1789116635) [ 1553.906296] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg633-server [ 1556.228101] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:50:40 (1789116640) [ 1558.625908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg633-server [ 1561.244695] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:50:45 (1789116645) [ 1563.089686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg633-server [ 1565.117885] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:50:49 (1789116649) [ 1567.539789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg633-server [ 1569.213370] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:50:54 (1789116654) [ 1571.586847] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg633-server [ 1573.522168] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:50:58 (1789116658) [ 1575.655365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg633-server [ 1578.036805] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:51:02 (1789116662) [ 1579.802217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg633-server [ 1581.996797] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:51:06 (1789116666) [ 1584.044301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg633-server [ 1586.214758] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:51:10 (1789116670) [ 1588.448301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg633-server [ 1590.231109] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:51:15 (1789116675) [ 1590.483785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.536138] LNet: 82624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.564252] LNetError: 82624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.581132] LNet: Removed LNI 192.168.206.33@tcp [ 1591.654390] LNet: Removed LNI 192.168.206.33@tcp1 [ 1591.718775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.794991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1591.838621] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1592.870519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1592.985707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1593.257208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.285378] LNet: 82771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.302094] LNetError: 82771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.314132] LNetError: 82771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1593.333218] LNet: Removed LNI 192.168.206.33@tcp [ 1593.404179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.477855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1593.520590] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1594.500784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1594.728388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.768765] LNetError: 82916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.798065] LNet: Removed LNI 192.168.206.33@tcp [ 1594.870994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.954569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.988401] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1595.888128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1595.993863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1596.228883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.284158] LNet: 83064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.300209] LNet: 83064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1597.349958] LNet: Removed LNI 192.168.206.33@tcp [ 1597.363847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.407926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1597.436707] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1598.558357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1598.814624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.884958] LNetError: 83209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1598.897180] LNetError: 83209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1598.912838] LNet: Removed LNI 192.168.206.33@tcp [ 1598.942452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.027465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.062855] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1599.831594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1599.913445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1600.127053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.196084] LNet: Removed LNI 192.168.206.33@tcp [ 1601.235818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.337043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1601.360176] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1602.131953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1602.487849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.527898] LNet: 83501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.534365] LNet: 83501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1603.556872] LNet: Removed LNI 192.168.206.33@tcp [ 1603.609543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.693400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1603.726749] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1604.657543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1604.751924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1605.026070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.049786] LNetError: 83649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.055853] LNetError: 83649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1605.076823] LNet: Removed LNI 192.168.206.33@tcp [ 1605.116519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.171673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1605.207673] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1606.034710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1606.307887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.397815] LNet: Removed LNI 192.168.206.33@tcp [ 1607.432907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.494353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1607.522663] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1608.322145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1608.404368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1608.736283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.832951] LNet: Removed LNI 192.168.206.33@tcp [ 1609.888926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.003655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1610.040365] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1610.691507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1610.926149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.956081] LNet: 84087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.964663] LNet: 84087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1612.005364] LNet: Removed LNI 192.168.206.33@tcp [ 1612.064672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.137900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.163520] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1613.034329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1613.099536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1613.394829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1613.421649] LNetError: 84234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1613.431674] LNetError: 84234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1613.458199] LNet: Removed LNI 192.168.206.33@tcp [ 1613.514032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1613.603360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1613.650033] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1614.415849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1615.841207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.934887] LNet: Removed LNI 192.168.206.33@tcp [ 1616.971938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.071798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.117409] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1618.100980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1618.203696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1618.273369] LNet: Removed LNI 192.168.206.33@tcp [ 1618.298369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.389442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.422929] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1619.321773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1619.441621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.488487] LNet: Removed LNI 192.168.206.33@tcp [ 1619.520891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.588946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1619.706534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.767537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.850684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1619.920462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1619.951716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1619.998552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1620.097941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.151791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.190420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1620.211873] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1621.065232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1621.118626] LNet: Removed LNI 192.168.206.33@tcp [ 1621.137391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1621.201752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1629.549542] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:51:53 (1789116713) [ 1629.672265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.700198] LNet: 85404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.706265] LNet: 85404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1629.744703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.951921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1630.001814] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1631.290427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.335556] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.340621] LNetError: 85549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1631.359367] LNet: Removed LNI 192.168.206.33@tcp231 [ 1631.377366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.427728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1631.446077] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1631.450468] LNet: 85557:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.873787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1631.897923] LNet: Removed LNI 192.168.206.33@tcp231 [ 1631.909704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1631.931717] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1633.257624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.293756] LNet: Removed LNI 192.168.206.33@tcp231 [ 1633.323685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.385573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1633.405336] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/0] [ 1633.409405] LNet: 85709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1633.946227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1633.987195] LNet: Removed LNI 192.168.206.33@tcp231 [ 1634.015790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1634.056611] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/0] [ 1635.233735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.267147] LNet: Removed LNI 192.168.206.33@tcp231 [ 1635.295626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.376395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1635.408683] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/60] [ 1635.416648] LNet: 85861:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.876755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1635.901413] LNet: Removed LNI 192.168.206.33@tcp231 [ 1635.916540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1635.938753] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/60] [ 1637.031666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.061500] LNet: Removed LNI 192.168.206.33@tcp231 [ 1637.086436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1637.177061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1637.210193] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1637.455780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1637.485806] LNet: Removed LNI 192.168.206.33@tcp231 [ 1637.497927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1637.517694] LNet: Added LNI 192.168.206.33@tcp231 [8/256/0/180] [ 1648.457754] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:52:12 (1789116732) [ 1648.752281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.852321] LNet: Removed LNI 192.168.206.33@tcp231 [ 1648.896980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.988096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1649.024810] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1650.010581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1650.035549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1650.060539] LNet: Removed LNI 192.168.206.33@tcp232 [ 1650.073636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1650.121151] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1651.080859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1651.115359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1651.141249] LNet: Removed LNI 192.168.206.33@tcp232 [ 1651.162987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1651.192781] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1652.207906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1652.267571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1652.322797] LNet: Removed LNI 192.168.206.33@tcp232 [ 1652.362561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1652.419489] LNet: Added LNI 192.168.206.33@tcp232 [8/256/0/180] [ 1653.486921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1653.561424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1653.604229] LNet: Removed LNI 192.168.206.33@tcp232 [ 1653.627314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1663.002306] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:52:27 (1789116747) [ 1663.723552] LNet: 87852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1663.730672] LNet: 87852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1664.795412] Key type .llcrypt unregistered [ 1664.802187] Key type ._llcrypt unregistered [ 1672.466582] Key type ._llcrypt registered [ 1672.470736] Key type .llcrypt registered [ 1672.668626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1672.690881] alg: No test for adler32 (adler32-zlib) [ 1673.701352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1673.763231] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1682.331962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1691.070266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.124505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1691.163448] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1692.763186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_interrupt -r 2 [ 1692.795955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 2 [ 1692.833331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_interrupt -r 2 [ 1692.890730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 2 [ 1692.926335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_interrupt -r 2 [ 1692.990449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_interrupt -r 2 [ 1693.053556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_interrupt -r 2 [ 1693.091463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_interrupt -r 2 [ 1693.128469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1693.152769] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1693.159207] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1693.505464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1693.538797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1693.570329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1693.599600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1693.622323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1693.653703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1693.696457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.754612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.783948] LNet: 88520:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.821882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.865378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.899679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.930236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1693.989751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1694.059701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1694.822475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_dropped -r 2 [ 1694.868915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 2 [ 1694.913752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_dropped -r 2 [ 1694.963063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 2 [ 1695.026222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_dropped -r 2 [ 1695.100707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_dropped -r 2 [ 1695.164938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_dropped -r 2 [ 1695.204537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_dropped -r 2 [ 1695.254070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1695.312500] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1695.637729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1695.665679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1695.691232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1695.722835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1695.760545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1695.811669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1695.860361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1695.884515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1695.920930] LNet: 88640:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.969937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.053436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.119624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.175565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.215636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.261326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1696.926697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_aborted -r 2 [ 1696.962413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 2 [ 1697.034078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_aborted -r 2 [ 1697.094986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 2 [ 1697.151736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_aborted -r 2 [ 1697.219404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_aborted -r 2 [ 1697.284991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_aborted -r 2 [ 1697.330596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_aborted -r 2 [ 1697.374580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1697.399580] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1697.690050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1697.716446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1697.748731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1697.794567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1697.831664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1697.868566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1697.921394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1697.960530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1697.988151] LNet: 88761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1698.007543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1698.068948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1698.107992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1698.192915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1698.267205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1698.316294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1699.050188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_no_route -r 2 [ 1699.103827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 2 [ 1699.164196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_no_route -r 2 [ 1699.203353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 2 [ 1699.260181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_no_route -r 2 [ 1699.332839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_no_route -r 2 [ 1699.408373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_no_route -r 2 [ 1699.456588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_no_route -r 2 [ 1699.502315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1699.540970] LNet: There was an unexpected network error while writing to 192.168.206.133: rc = -22 [ 1699.809933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1699.850979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1699.875453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1699.902974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1699.933852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1699.960630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1699.988304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1700.019705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1700.040243] LNet: 88881:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1700.067588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1700.103442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1700.149556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1700.173722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1700.205761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1700.229654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1700.840745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e local_timeout -r 2 [ 1700.890440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 2 [ 1700.953520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e local_timeout -r 2 [ 1701.009318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 2 [ 1701.057569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e local_timeout -r 2 [ 1701.105360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e local_timeout -r 2 [ 1701.155415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e local_timeout -r 2 [ 1701.199979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e local_timeout -r 2 [ 1701.234188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1701.518964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1701.562861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1701.614990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1701.672820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1701.715629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1701.743162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1701.770318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1701.808872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1701.839519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1701.867904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1701.904931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1701.941583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1701.981294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1702.015807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1702.577210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp -m GET -e remote_dropped -r 2 [ 1702.617522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 2 [ 1702.666868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp -m GET -e remote_dropped -r 2 [ 1702.724377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 2 [ 1702.771657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp -m GET -e remote_dropped -r 2 [ 1702.826849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.133@tcp1 -m GET -e remote_dropped -r 2 [ 1702.876551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp -m GET -e remote_dropped -r 2 [ 1702.939711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.33@tcp1 -d 192.168.206.33@tcp1 -m GET -e remote_dropped -r 2 [ 1702.985548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.33@tcp 192.168.206.133@tcp [ 1703.016551] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 1703.283403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1703.318546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1703.387385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1703.408871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1703.433545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1703.480569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1703.528165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1703.575433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1703.608305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1703.651569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1703.677890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1703.715429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1703.764052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1703.797830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1709.697876] LNet: 89200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.706816] LNetError: 89200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.718281] LNet: Removed LNI 192.168.206.33@tcp [ 1709.733365] LNet: Removed LNI 192.168.206.33@tcp1 [ 1710.145703] Key type .llcrypt unregistered [ 1710.150775] Key type ._llcrypt unregistered [ 1728.578452] Key type ._llcrypt registered [ 1728.580324] Key type .llcrypt registered [ 1729.895546] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:53:34 (1789116814) [ 1730.349487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1730.376488] alg: No test for adler32 (adler32-zlib) [ 1731.285445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.347707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.387430] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1732.323513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1732.351693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.33@tcp [ 1732.376859] LNetError: 90179:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1732.386082] LNet: 1 local NIs in recovery (showing 1): 192.168.206.33@tcp [ 1732.493718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.33@tcp [ 1732.609232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1739.900872] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:53:44 (1789116824) [ 1740.022527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.082401] LNet: 90766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.087503] LNetError: 90766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.118714] LNet: Removed LNI 192.168.206.33@tcp [ 1740.183486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1740.277573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1740.301280] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1741.129626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1741.176538] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 1742.441710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.33@tcp [ 1742.484964] LNetError: 91037:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.33@tcp to recovery [ 1743.330880] LNet: There was an unexpected network error while writing to 192.168.206.33: rc = -22 [ 1743.759135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.33@tcp [ 1752.740857] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:53:57 (1789116837) [ 1755.017224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg633-server [ 1757.334228] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:54:01 (1789116841) [ 1759.268696] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg633-server [ 1761.261100] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:54:05 (1789116845) [ 1763.339126] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg633-server [ 1765.115324] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:54:10 (1789116850) [ 1765.230932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.262287] LNet: 92662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.276656] LNetError: 92662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.293904] LNet: Removed LNI 192.168.206.33@tcp [ 1765.310194] LNet: Removed LNI 192.168.206.33@tcp1 [ 1765.346384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.454407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.481796] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1766.582074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1766.699728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1766.772366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1766.881209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1766.945456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1766.997512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1774.578950] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:54:19 (1789116859) [ 1774.670156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.719645] LNet: 93390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1774.733647] LNetError: 93390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1774.742252] LNetError: 93390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1774.778111] LNet: Removed LNI 192.168.206.33@tcp [ 1774.800148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.870625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1774.910274] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1783.319174] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:54:27 (1789116867) [ 1783.478860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.505604] LNet: 94091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.511409] LNetError: 94091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.529606] LNet: Removed LNI 192.168.206.33@tcp [ 1783.547984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.633078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.669296] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1793.236818] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:54:37 (1789116877) [ 1795.903602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1798.092414] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:54:42 (1789116882) [ 1798.570153] LNet: 95133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.583787] LNetError: 95133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.610471] LNet: Removed LNI 192.168.206.33@tcp [ 1799.589467] Key type .llcrypt unregistered [ 1799.594053] Key type ._llcrypt unregistered [ 1805.382407] Key type ._llcrypt registered [ 1805.384934] Key type .llcrypt registered [ 1805.583510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1805.597872] alg: No test for adler32 (adler32-zlib) [ 1806.588943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1806.637679] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1813.242729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 1818.728394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.133@tcp [ 1820.091398] LNet: 95560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.095461] LNetError: 95560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.105748] LNet: Removed LNI 192.168.206.33@tcp [ 1820.402892] Key type .llcrypt unregistered [ 1820.411612] Key type ._llcrypt unregistered [ 1830.624653] Key type ._llcrypt registered [ 1830.627733] Key type .llcrypt registered [ 1831.999292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1834.250441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1835.836972] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:55:20 (1789116920) [ 1837.532758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1839.328133] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:55:24 (1789116924) [ 1840.733218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg633-server [ 1842.325862] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:55:27 (1789116927) [ 1844.887443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg633-server [ 1847.160906] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:55:31 (1789116931) [ 1847.965245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.140089] alg: No test for adler32 (adler32-zlib) [ 1849.261368] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 1849.540516] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1851.303233] Key type lgssc registered [ 1852.729921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1878.055673] Key type lgssc unregistered [ 1878.460080] LNet: 98738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.470489] LNetError: 98738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.499713] LNet: Removed LNI 192.168.206.33@tcp [ 1879.156431] Key type .llcrypt unregistered [ 1879.161192] Key type ._llcrypt unregistered [ 1895.411193] Key type ._llcrypt registered [ 1895.415380] Key type .llcrypt registered [ 1896.897277] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:56:21 (1789116981) [ 1897.249738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.267462] alg: No test for adler32 (adler32-zlib) [ 1898.267716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.324942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1898.363371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1898.400210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1898.430822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1905.378154] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:56:30 (1789116990) [ 1905.496089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.518935] LNet: 100145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.545034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.661770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.697241] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1906.474106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.487203] LNet: 100296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.492158] LNetError: 100296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.511355] LNet: Removed LNI 192.168.206.33@tcp [ 1906.541324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.660804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1906.698534] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1907.316461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.343332] LNetError: 100325:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.361589] LNet: Removed LNI 192.168.206.33@tcp [ 1907.380414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.448044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1907.486641] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1913.904394] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:56:39 (1789116999) [ 1914.004204] LNet: 100909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.009777] LNet: 100909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1914.017939] LNetError: 100909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.036283] LNet: Removed LNI 192.168.206.33@tcp [ 1914.556942] Key type .llcrypt unregistered [ 1914.559197] Key type ._llcrypt unregistered [ 1920.049231] Key type ._llcrypt registered [ 1920.056131] Key type .llcrypt registered [ 1920.394446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1920.407711] alg: No test for adler32 (adler32-zlib) [ 1921.354571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.455619] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1921.459972] LNet: *** cfs_fail_loc=e003, val=100*** [ 1922.412849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1922.434306] LNet: *** cfs_fail_loc=e003, val=100*** [ 1922.437376] LNet: Skipped 2 previous similar messages [ 1922.453212] Ping reply received with mapping: [ 1922.454653] found metadata reply NI[192.168.206.33@tcp] [ 1922.458416] 21 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1922.468975] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1922.475450] 5a 5a 5a 5a 5a 5a 5a 5a [ 1922.649426] LNet: 101317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1922.671071] LNetError: 101317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1922.704085] LNet: Removed LNI 192.168.206.33@tcp [ 1923.437475] Key type .llcrypt unregistered [ 1923.447200] Key type ._llcrypt unregistered [ 1934.664940] Key type ._llcrypt registered [ 1934.666307] Key type .llcrypt registered [ 1935.984758] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:57:00 (1789117020) [ 1936.546440] Key type .llcrypt unregistered [ 1936.548259] Key type ._llcrypt unregistered [ 1943.322440] Key type ._llcrypt registered [ 1943.327784] Key type .llcrypt registered [ 1943.535361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1943.548521] alg: No test for adler32 (adler32-zlib) [ 1944.748532] LNetError: 102374:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1945.224574] Key type .llcrypt unregistered [ 1945.231962] Key type ._llcrypt unregistered [ 1950.899170] Key type ._llcrypt registered [ 1950.901020] Key type .llcrypt registered [ 1952.217518] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 04:57:16 (1789117036) [ 1952.822580] Key type .llcrypt unregistered [ 1952.826280] Key type ._llcrypt unregistered [ 1959.308640] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1959.317745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1962.131762] Key type ._llcrypt registered [ 1962.140222] Key type .llcrypt registered [ 1962.488569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1962.520773] alg: No test for adler32 (adler32-zlib) [ 1963.452968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.534373] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1975.105576] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 04:57:39 (1789117059) [ 1975.247793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.284559] LNet: 104157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1975.306366] LNetError: 104157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1975.342512] LNet: Removed LNI 10.1.2.3@tcp [ 1975.388248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.611851] Key type .llcrypt unregistered [ 1976.616352] Key type ._llcrypt unregistered [ 1983.902340] Key type ._llcrypt registered [ 1983.904054] Key type .llcrypt registered [ 1984.105952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1984.120725] alg: No test for adler32 (adler32-zlib) [ 1985.075227] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 1985.155250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 1985.339075] LNet: 104456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.354841] LNetError: 104456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1986.407067] LNet: Removed LNI 192.168.206.33@tcp [ 1987.370596] Key type .llcrypt unregistered [ 1987.382081] Key type ._llcrypt unregistered [ 1999.812329] Key type ._llcrypt registered [ 1999.815084] Key type .llcrypt registered [ 2001.308842] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 04:58:05 (1789117085) [ 2002.172250] Key type .llcrypt unregistered [ 2002.175184] Key type ._llcrypt unregistered [ 2008.381636] Key type ._llcrypt registered [ 2008.384286] Key type .llcrypt registered [ 2008.583871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.598372] alg: No test for adler32 (adler32-zlib) [ 2016.236476] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 04:58:21 (1789117101) [ 2016.365891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2016.393674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2016.455359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2016.480329] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2017.258402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2017.319970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2017.353111] LNet: Removed LNI 192.168.206.33@tcp [ 2017.624623] LNet: 106212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.090743] Key type .llcrypt unregistered [ 2018.096554] Key type ._llcrypt unregistered [ 2029.933078] Key type ._llcrypt registered [ 2029.934918] Key type .llcrypt registered [ 2031.132607] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 04:58:36 (1789117116) [ 2031.507388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2031.527937] alg: No test for adler32 (adler32-zlib) [ 2032.452309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.526832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2032.566368] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2033.268758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2033.297648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.33@tcp [ 2033.314236] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2033.318827] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.321473] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 0 [ 2033.325594] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2033.330984] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2033.334693] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.337618] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 0 [ 2033.342383] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2033.344476] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2033.347552] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.350419] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 2083 [ 2033.354297] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2033.356292] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2033.359254] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2033.361917] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2033, 0, 2083 [ 2033.365724] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2039.417959] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 04:58:44 (1789117124) [ 2040.032451] LNet: 107751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.045443] LNetError: 107751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2040.069988] LNet: Removed LNI 192.168.206.33@tcp [ 2040.734616] Key type .llcrypt unregistered [ 2040.736424] Key type ._llcrypt unregistered [ 2046.495741] Key type ._llcrypt registered [ 2046.499320] Key type .llcrypt registered [ 2046.760059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.782031] alg: No test for adler32 (adler32-zlib) [ 2047.821802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2047.872590] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2055.890339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 2064.852654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.925514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2064.987987] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 2066.273097] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2066.387598] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2066.396969] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2066.408652] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2068.972825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2069.006360] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2083.196238] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 04:59:28 (1789117168) [ 2083.966251] LNet: 109263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.981908] LNetError: 109263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2084.008094] LNet: Removed LNI 192.168.206.33@tcp [ 2085.025060] LNet: Removed LNI 192.168.206.33@tcp1 [ 2085.033147] LNetError: 109263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2085.060668] LNetError: 109263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2085.151340] LNet: Removed LNI 10.1.2.3@tcp99 [ 2086.643681] Key type .llcrypt unregistered [ 2086.647976] Key type ._llcrypt unregistered [ 2093.758051] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2093.766329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2096.387185] Key type ._llcrypt registered [ 2096.396153] Key type .llcrypt registered [ 2096.565887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2096.581287] alg: No test for adler32 (adler32-zlib) [ 2097.454781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.498944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2097.529526] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2098.311278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2098.355783] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2101.388547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2101.474320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2101.532305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2101.565646] LNetError: 110012:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2101.598968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2101.660967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2102.096305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2102.193388] LNet: 110032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.205216] LNetError: 110032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2102.215328] LNet: Removed LNI 10.1.2.3@tcp [ 2103.275884] LNet: Removed LNI 192.168.206.33@tcp [ 2103.883762] Key type .llcrypt unregistered [ 2103.887734] Key type ._llcrypt unregistered [ 2116.347709] Key type ._llcrypt registered [ 2116.354703] Key type .llcrypt registered [ 2117.981573] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:00:02 (1789117202) [ 2118.439800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2118.460405] alg: No test for adler32 (adler32-zlib) [ 2119.476869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.575735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.618054] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2120.570933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2120.688948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg633-client@tcp [ 2128.912970] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:00:13 (1789117213) [ 2129.042612] LNet: 111561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.065888] LNetError: 111561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.109046] LNet: Removed LNI 192.168.206.33@tcp [ 2129.920591] Key type .llcrypt unregistered [ 2129.922278] Key type ._llcrypt unregistered [ 2137.594787] Key type ._llcrypt registered [ 2137.599261] Key type .llcrypt registered [ 2137.710250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2137.724929] alg: No test for adler32 (adler32-zlib) [ 2138.737554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2138.774897] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2138.866424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2145.886245] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:00:31 (1789117231) [ 2147.679489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2149.276514] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:00:34 (1789117234) [ 2151.149453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2152.723147] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:00:37 (1789117237) [ 2152.912745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.943654] LNet: 112991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.954838] LNetError: 112991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.005454] LNet: Removed LNI 192.168.206.33@tcp [ 2153.032824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.093320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.120415] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2153.727477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2153.772314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2161.176680] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:00:45 (1789117245) [ 2161.267727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.293921] LNet: 113698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.300627] LNetError: 113698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.316223] LNet: Removed LNI 192.168.206.33@tcp [ 2161.335527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.393719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2161.452082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2162.304271] Key type .llcrypt unregistered [ 2162.307650] Key type ._llcrypt unregistered [ 2174.419123] Key type ._llcrypt registered [ 2174.422396] Key type .llcrypt registered [ 2175.881943] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:01:00 (1789117260) [ 2176.362173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2176.382270] alg: No test for adler32 (adler32-zlib) [ 2177.420691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.507338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2177.555349] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2178.491898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2178.589675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2185.645110] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:01:10 (1789117270) [ 2186.039076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.079520] LNet: 115279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.094726] LNetError: 115279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2186.123454] LNet: Removed LNI 192.168.206.33@tcp [ 2186.149460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.209766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2186.249401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2193.353085] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:01:18 (1789117278) [ 2193.475842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.515526] LNet: 115853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.536412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.612683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2194.032111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2194.070840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2194.155979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2194.672523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2194.710810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2194.763407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.778764] LNet: 115892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.795851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.845356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2195.152548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2195.180577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2195.254597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2195.496296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2195.519838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2204.939431] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:01:28 (1789117288) [ 2205.097610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2205.166859] LNet: 116484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2205.233489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.353612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.407875] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2206.510563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2206.540264] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2207.652731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2207.776858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2208.216981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.251473] LNetError: 116772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.259241] LNet: Removed LNI 192.168.206.33@tcp [ 2208.270264] LNet: Removed LNI 192.168.206.33@tcp2 [ 2208.297831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.368729] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2208.373835] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2208.380323] LNet: 116778:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2208.981241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2209.238227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.262775] LNet: 116815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.270665] LNet: 116815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2209.282405] LNetError: 116815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.287787] LNetError: 116815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2209.295319] LNet: Removed LNI 192.168.206.33@tcp [ 2209.304981] LNet: Removed LNI 192.168.206.33@tcp2 [ 2209.319801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.358250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2209.378779] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2210.262831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2210.279190] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2212.008234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2212.066904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2212.188866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2212.538648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2212.574389] LNetError: 117100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.583283] LNetError: 117100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2212.601210] LNet: Removed LNI 192.168.206.33@tcp [ 2213.672135] LNet: Removed LNI 192.168.206.33@tcp2 [ 2213.748758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2213.871397] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2213.881188] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2213.892890] LNet: 117106:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2214.585983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2214.797908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2214.830821] LNet: Removed LNI 192.168.206.33@tcp [ 2215.851508] LNet: Removed LNI 192.168.206.33@tcp2 [ 2215.877910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.981564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2216.033322] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2217.018964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2217.039120] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2218.099928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2218.177648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2218.545312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2218.591988] LNet: 117423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.599972] LNet: 117423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2218.606648] LNetError: 117423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.613656] LNetError: 117423:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2218.621136] LNet: Removed LNI 192.168.206.33@tcp [ 2218.631953] LNet: Removed LNI 192.168.206.33@tcp2 [ 2218.652519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2218.745484] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2218.748676] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2218.758869] LNet: 117429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.708808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2219.980736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2221.023145] LNet: Removed LNI 192.168.206.33@tcp [ 2221.034029] LNet: Removed LNI 192.168.206.33@tcp2 [ 2221.067346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.213340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.252943] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2222.327992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2222.380903] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2223.469706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2223.535546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2223.627350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2223.951749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2224.991659] LNet: Removed LNI 192.168.206.33@tcp [ 2226.022133] LNet: Removed LNI 192.168.206.33@tcp2 [ 2226.060418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.131904] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2226.142042] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2226.147703] LNet: 117757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2226.657983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2226.849947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.874953] LNetError: 117794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2226.882763] LNetError: 117794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2226.890740] LNet: Removed LNI 192.168.206.33@tcp [ 2227.943829] LNet: Removed LNI 192.168.206.33@tcp2 [ 2227.969753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.033179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.087613] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2228.787813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2228.804346] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2229.623480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2229.657960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2229.726588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2230.085568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.135163] LNet: Removed LNI 192.168.206.33@tcp [ 2231.145585] LNet: Removed LNI 192.168.206.33@tcp2 [ 2231.193583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.274432] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2231.285863] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2231.290629] LNet: 118096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2231.834242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2232.202522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2233.247228] LNet: Removed LNI 192.168.206.33@tcp [ 2233.260333] LNet: Removed LNI 192.168.206.33@tcp2 [ 2233.295414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.388622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2233.450867] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2234.171705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2234.192839] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2235.130838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2235.180220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2235.218134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2235.282195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2235.552477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2235.567641] LNet: 118450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.571778] LNet: 118450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2236.639162] LNet: Removed LNI 192.168.206.33@tcp [ 2237.672740] LNet: Removed LNI 192.168.206.33@tcp2 [ 2237.712802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.799793] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2237.804832] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2238.147184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2245.123995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2246.705322] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:02:11 (1789117331) [ 2247.021900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.059220] LNetError: 119216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2247.068900] LNetError: 119216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2247.075558] LNet: Removed LNI 192.168.206.33@tcp [ 2248.116867] LNet: Removed LNI 192.168.206.33@tcp2 [ 2248.202955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.672664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.698800] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2249.698781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2249.744890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2250.707425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.745780] LNet: Removed LNI 10.1.2.3@tcp [ 2250.758206] LNet: Removed LNI 192.168.206.33@tcp [ 2250.782976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.846283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2250.877977] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2250.890754] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.293844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2251.310846] LNet: Removed LNI 192.168.206.33@tcp [ 2251.338977] LNet: Removed LNI 10.1.2.3@tcp [ 2259.487961] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 05:02:23 (1789117343) [ 2259.828508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.950947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.043708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.105401] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2261.016069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2261.063180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.096805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.124880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.150864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.189746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.230726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.291398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.364851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.426456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.488854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.33@tcp [ 2261.706230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2270.833367] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:02:35 (1789117355) [ 2270.973602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.010144] LNet: 120824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.017981] LNet: 120824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2271.051908] LNet: Removed LNI 192.168.206.33@tcp [ 2271.081593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.261604] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2271.431040] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2271.439585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2271.459358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2273.793156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.809708] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2274.547579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2274.570524] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2276.913820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2276.938534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2276.983408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.33@tcp -m PUT -e local_timeout -r 1 [ 2277.040766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2277.085296] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2277.090616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2277.373442] LNetError: 120898:0:(lib-move.c:4801:lnet_parse()) 192.168.206.33@tcp, src 192.168.206.33@tcp: Dropping PUT (error -108 looking up sender) [ 2278.111967] LNet: Removed LNI 10.1.2.3@tcp [ 2279.157751] LNet: Removed LNI 192.168.206.33@tcp [ 2279.568175] Key type .llcrypt unregistered [ 2279.572378] Key type ._llcrypt unregistered [ 2292.173595] Key type ._llcrypt registered [ 2292.175851] Key type .llcrypt registered [ 2293.292940] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:02:58 (1789117378) [ 2293.883372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2293.903136] alg: No test for adler32 (adler32-zlib) [ 2294.818570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.257617] LNet: 122213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.847850] Key type .llcrypt unregistered [ 2295.851695] Key type ._llcrypt unregistered [ 2301.685580] Key type ._llcrypt registered [ 2301.689871] Key type .llcrypt registered [ 2301.886775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2301.902294] alg: No test for adler32 (adler32-zlib) [ 2302.812317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2302.840388] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2309.687333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 2309.862723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2310.529811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.133@tcp [ 2310.765779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2310.801181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2310.829325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2310.853963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2310.878957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2310.910322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2310.946597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2310.980584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2311.000164] LNet: 122668:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2311.027702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2311.073498] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.133@tcp [ 2311.086151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2311.126445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2311.154879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2311.187786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2311.227473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2311.814151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2312.003383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2312.030735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2312.057818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2312.086086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2312.117881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2312.147294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2312.182852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2312.213230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2312.228514] LNet: 122785:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2312.243711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2312.287523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2312.332475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2312.388822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2312.424858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2312.459895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2313.022052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.133@tcp [ 2313.256248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2313.281036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2313.301205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2313.327201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2313.368512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2313.397492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2313.443721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2313.480238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2313.497454] LNet: 122902:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2313.513837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2313.537382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2313.575145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2313.621875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2313.652813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2313.676337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2314.250110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2314.434627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2314.456720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2314.486962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2314.529350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2314.561447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2314.597622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2314.638413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2314.672330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2314.704677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2314.740769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2314.771784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2314.805950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2314.843946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2314.873433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2315.192814] LNet: 123069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.200424] LNetError: 123069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2315.214262] LNet: Removed LNI 192.168.206.33@tcp [ 2315.469744] Key type .llcrypt unregistered [ 2315.473314] Key type ._llcrypt unregistered [ 2329.887560] Key type ._llcrypt registered [ 2329.892322] Key type .llcrypt registered [ 2331.143721] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:03:36 (1789117416) [ 2331.490852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2331.513460] alg: No test for adler32 (adler32-zlib) [ 2332.467557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2332.832734] LNet: 123957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.502365] Key type .llcrypt unregistered [ 2333.504575] Key type ._llcrypt unregistered [ 2339.571306] Key type ._llcrypt registered [ 2339.576470] Key type .llcrypt registered [ 2339.837160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2339.853777] alg: No test for adler32 (adler32-zlib) [ 2340.764496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2340.803875] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2347.975572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.133@tcp [ 2356.120595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.164274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2356.191958] LNet: Added LNI 192.168.206.33@tcp1 [8/256/0/180] [ 2357.008443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2357.581910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.133@tcp [ 2357.796918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2357.834718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2357.871133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2357.906281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2357.936882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2357.973374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2358.000785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2358.037540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2358.052529] LNet: 124613:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.071973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2358.101349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2358.146389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2358.188438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2358.206773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2358.228879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2358.749676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2358.979735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2359.010142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2359.048180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2359.082476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2359.133286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2359.166209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2359.198897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2359.230143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2359.255206] LNet: 124730:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.263819] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.133@tcp, 192.168.206.133@tcp1 [ 2359.279174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2359.308759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2359.344659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2359.380724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2359.416755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2359.456694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2359.969487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.133@tcp [ 2360.135440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2360.166111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2360.200928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2360.236795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2360.293745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2360.326555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2360.368775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2360.417716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2360.436023] LNet: 124848:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2360.472740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2360.510424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2360.540544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2360.573258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2360.600883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2360.628334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2361.194340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2361.399965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2361.432952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2361.472996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2361.502278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2361.538344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2361.581603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2361.619464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2361.643245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2361.673885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2361.699803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2361.720933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2361.747768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2361.776502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2361.818037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2366.992240] LNet: 125049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.013731] LNetError: 125049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.031206] LNet: Removed LNI 192.168.206.33@tcp [ 2368.033282] LNetError: 125049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.065717] LNet: Removed LNI 192.168.206.33@tcp1 [ 2368.589863] Key type .llcrypt unregistered [ 2368.592349] Key type ._llcrypt unregistered [ 2385.033486] Key type ._llcrypt registered [ 2385.036456] Key type .llcrypt registered [ 2386.287854] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 05:04:31 (1789117471) [ 2386.958337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2386.976151] alg: No test for adler32 (adler32-zlib) [ 2387.853475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.911504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2387.950104] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2388.559349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2388.575536] LNet: Added LNI 192.168.206.33@tcp2 [8/256/0/180] [ 2389.502387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.33@tcp [ 2389.604249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2390.687214] LNet: Removed LNI 192.168.206.33@tcp [ 2396.993792] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 05:04:42 (1789117482) [ 2398.487569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2399.975708] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:04:45 (1789117485) [ 2406.986958] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:04:51 (1789117491) [ 2413.407931] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:04:58 (1789117498) [ 2420.330953] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:05:05 (1789117505) [ 2427.698425] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:05:12 (1789117512) [ 2435.870490] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:05:20 (1789117520) [ 2444.474585] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:05:29 (1789117529) [ 2451.881872] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:05:36 (1789117536) [ 2460.337660] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:05:45 (1789117545) [ 2467.908698] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:05:53 (1789117553) [ 2474.602114] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:05:59 (1789117559) [ 2480.938807] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:06:06 (1789117566) [ 2487.402822] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:06:12 (1789117572) [ 2493.401553] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:06:18 (1789117578) [ 2499.981581] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:06:25 (1789117585) [ 2507.388496] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:06:32 (1789117592) [ 2514.552933] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:06:39 (1789117599) [ 2521.863330] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:06:46 (1789117606) [ 2522.110768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2522.130046] LNet: 136748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2522.153498] LNetError: 136748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2522.188098] LNet: Removed LNI 192.168.206.33@tcp2 [ 2522.233374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.388372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2522.419592] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2524.089088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2524.106610] LNet: 136890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2524.114259] LNetError: 136890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2524.129723] LNet: Removed LNI 192.168.206.33@tcp [ 2524.162523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2524.242420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2524.281734] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2532.045652] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:06:56 (1789117616) [ 2539.409447] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:07:04 (1789117624) [ 2545.794552] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:07:10 (1789117630) [ 2553.657462] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:07:18 (1789117638) [ 2562.508639] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:07:26 (1789117646) [ 2571.017926] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:07:36 (1789117656) [ 2578.994375] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:07:43 (1789117663) [ 2587.111417] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:07:52 (1789117672) [ 2595.427299] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:08:00 (1789117680) [ 2603.071728] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:08:08 (1789117688) [ 2611.848955] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:08:16 (1789117696) [ 2619.291415] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:08:24 (1789117704) [ 2626.667994] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2420 sec ========== 05:08:31 (1789117711) [ 2627.053314] LNet: 144359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2627.059312] LNetError: 144359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2627.075201] LNet: Removed LNI 192.168.206.33@tcp [ 2627.754699] Key type .llcrypt unregistered [ 2627.770432] Key type ._llcrypt unregistered [ 2641.616597] Key type ._llcrypt registered [ 2641.657601] Key type .llcrypt registered [ 2642.043896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2642.085452] alg: No test for adler32 (adler32-zlib) [ 2643.171075] Lustre: Lustre: Build Version: 2.17.57_142_g748fc9f [ 2643.422541] LNet: Added LNI 192.168.206.33@tcp [8/256/0/180] [ 2645.112902] Key type lgssc registered [ 2646.306494] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2716.491734] Lustre: Mounted lustre-client - version 2.17.57_142_g748fc9f [ 2721.526418] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2728.490585] Lustre: Unmounted lustre-client [ 2757.353847] Key type lgssc unregistered [ 2757.575084] LNet: 147329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2757.593320] LNetError: 147329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2757.620721] LNet: Removed LNI 192.168.206.33@tcp [ 2758.193451] Key type .llcrypt unregistered [ 2758.197937] Key type ._llcrypt unregistered