[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 466497797 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003346] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008030] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.011031] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.014018] *** VALIDATE selinux *** [ 0.021852] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025743] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028115] *** VALIDATE tmpfs *** [ 0.030209] *** VALIDATE proc *** [ 0.031301] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.034182] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036064] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038026] Spectre V2 : User space: Vulnerable [ 0.039000] Speculative Store Bypass: Vulnerable [ 0.040796] debug: unmapping init [mem 0xffffffffb9659000-0xffffffffb9660fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043610] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044025] ... version: 2 [ 0.045012] ... bit width: 48 [ 0.046009] ... generic registers: 4 [ 0.047010] ... value mask: 0000ffffffffffff [ 0.048007] ... max period: 00007fffffffffff [ 0.049009] ... fixed-purpose events: 3 [ 0.050006] ... event mask: 000000070000000f [ 0.052206] rcu: Hierarchical SRCU implementation. [ 0.054489] smp: Bringing up secondary CPUs ... [ 0.055517] x86: Booting SMP configuration: [ 0.056022] .... node #0, CPUs: #1 #2 #3 [ 0.061670] smp: Brought up 1 node, 4 CPUs [ 0.063020] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.122898] node 0 deferred pages initialised in 55ms [ 0.125427] devtmpfs: initialized [ 0.127409] x86/mm: Memory block size: 128MB [ 0.130825] gcov: version magic: 0x41383552 [ 0.133292] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139291] pinctrl core: initialized pinctrl subsystem [ 0.141159] [ 0.141717] ************************************************************* [ 0.144013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.147012] ** ** [ 0.149012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.152012] ** ** [ 0.154012] ** This means that this kernel is built to expose internal ** [ 0.157012] ** IOMMU data structures, which may compromise security on ** [ 0.159012] ** your system. ** [ 0.162009] ** ** [ 0.163013] ** If you see this message and you are not debugging the ** [ 0.165014] ** kernel, report this immediately to your vendor! ** [ 0.167013] ** ** [ 0.168009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170010] ************************************************************* [ 0.173584] NET: Registered protocol family 16 [ 0.175359] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178054] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.180038] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.184212] cpuidle: using governor menu [ 0.185828] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187406] PCI: Using configuration type 1 for base access [ 0.189100] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.194113] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199189] cryptd: max_cpu_qlen set to 1000 [ 0.202233] ACPI: Added _OSI(Module Device) [ 0.203006] ACPI: Added _OSI(Processor Device) [ 0.205009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210277] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.214663] ACPI: Interpreter enabled [ 0.216047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218007] ACPI: Using IOAPIC for interrupt routing [ 0.219093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224253] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.233890] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.236032] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.242058] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247200] acpiphp: Slot [2] registered [ 0.248119] acpiphp: Slot [5] registered [ 0.250102] acpiphp: Slot [6] registered [ 0.252101] acpiphp: Slot [3] registered [ 0.253099] acpiphp: Slot [4] registered [ 0.255074] acpiphp: Slot [7] registered [ 0.256213] acpiphp: Slot [8] registered [ 0.258071] acpiphp: Slot [9] registered [ 0.260149] acpiphp: Slot [10] registered [ 0.261088] acpiphp: Slot [11] registered [ 0.263088] acpiphp: Slot [12] registered [ 0.265074] acpiphp: Slot [13] registered [ 0.266074] acpiphp: Slot [14] registered [ 0.268078] acpiphp: Slot [15] registered [ 0.270071] acpiphp: Slot [16] registered [ 0.271083] acpiphp: Slot [17] registered [ 0.273106] acpiphp: Slot [18] registered [ 0.275134] acpiphp: Slot [19] registered [ 0.277095] acpiphp: Slot [20] registered [ 0.278089] acpiphp: Slot [21] registered [ 0.280073] acpiphp: Slot [22] registered [ 0.281076] acpiphp: Slot [23] registered [ 0.282078] acpiphp: Slot [24] registered [ 0.284070] acpiphp: Slot [25] registered [ 0.285074] acpiphp: Slot [26] registered [ 0.287130] acpiphp: Slot [27] registered [ 0.288048] acpiphp: Slot [28] registered [ 0.289065] acpiphp: Slot [29] registered [ 0.291086] acpiphp: Slot [30] registered [ 0.292100] acpiphp: Slot [31] registered [ 0.294050] PCI host bridge to bus 0000:00 [ 0.295011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.299015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.304018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.306022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.307138] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.310065] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.314000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.322022] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.326059] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.333016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.335014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.338759] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343079] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.345039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.347824] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350827] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.361015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.365873] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.370232] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.385020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.392014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.404015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.412814] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.419012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.425014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.445022] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.459961] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.462319] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.464340] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.466439] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.469152] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.474019] iommu: Default domain type: Passthrough [ 0.475596] SCSI subsystem initialized [ 0.476242] ACPI: bus type USB registered [ 0.477132] usbcore: registered new interface driver usbfs [ 0.479075] usbcore: registered new interface driver hub [ 0.481092] usbcore: registered new device driver usb [ 0.482162] pps_core: LinuxPPS API ver. 1 registered [ 0.484007] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.486052] PTP clock support registered [ 0.490163] EDAC MC: Ver: 3.0.0 [ 0.500173] PCI: Using ACPI for IRQ routing [ 0.503469] NetLabel: Initializing [ 0.505009] NetLabel: domain hash size = 128 [ 0.507008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.509077] NetLabel: unlabeled traffic allowed by default [ 0.511124] vgaarb: loaded [ 0.512506] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.515014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.523628] clocksource: Switched to clocksource kvm-clock [ 0.635942] VFS: Disk quotas dquot_6.6.0 [ 0.637646] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.641435] *** VALIDATE ramfs *** [ 0.642671] *** VALIDATE hugetlbfs *** [ 0.644202] pnp: PnP ACPI init [ 0.646492] pnp: PnP ACPI: found 6 devices [ 0.662585] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.666631] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.669090] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.671287] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.674083] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.676864] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.679771] NET: Registered protocol family 2 [ 0.682378] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.687565] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.691484] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.697259] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.699651] TCP: Hash tables configured (established 65536 bind 65536) [ 0.701539] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.703575] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705981] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.707959] NET: Registered protocol family 1 [ 0.710255] RPC: Registered named UNIX socket transport module. [ 0.712597] RPC: Registered udp transport module. [ 0.714468] RPC: Registered tcp transport module. [ 0.715649] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.718267] NET: Registered protocol family 44 [ 0.719967] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.721394] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.722852] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.724972] PCI: CLS 0 bytes, default 64 [ 0.727606] Unpacking initramfs... [ 2.225328] debug: unmapping init [mem 0xffff9778bcc64000-0xffff9778bffcffff] [ 2.228890] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.230856] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.233505] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.772089] Initialise system trusted keyrings [ 2.773892] Key type blacklist registered [ 2.777219] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.806107] zbud: loaded [ 2.811056] *** VALIDATE nfs *** [ 2.812273] *** VALIDATE nfs4 *** [ 2.814081] pstore: using deflate compression [ 2.823133] Platform Keyring initialized [ 2.941330] NET: Registered protocol family 38 [ 2.942617] Key type asymmetric registered [ 2.943760] Asymmetric key parser 'x509' registered [ 2.945507] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.948223] io scheduler mq-deadline registered [ 2.950167] io scheduler kyber registered [ 2.951704] io scheduler bfq registered [ 2.953873] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.956316] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.958595] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.960947] ACPI: Power Button [PWRF] [ 2.965407] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.971379] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.980487] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.007969] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.038316] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.044719] Non-volatile memory driver v1.3 [ 3.046390] Linux agpgart interface v0.103 [ 3.076929] virtio_blk virtio1: [vda] 134848 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.080068] vda: detected capacity change from 0 to 69042176 [ 3.112389] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.116487] vdb: detected capacity change from 0 to 1073741824 [ 3.134546] libphy: Fixed MDIO Bus: probed [ 3.141841] usbcore: registered new interface driver usbserial_generic [ 3.155460] usbserial: USB Serial support registered for generic [ 3.158256] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.163481] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.167128] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.170175] mousedev: PS/2 mouse device common for all mice [ 3.173335] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.176756] rtc_cmos 00:05: RTC can wake from S4 [ 3.181726] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.182983] rtc_cmos 00:05: registered as rtc0 [ 3.192296] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.196325] intel_pstate: CPU model not supported [ 3.200226] hid: raw HID events driver (C) Jiri Kosina [ 3.203464] usbcore: registered new interface driver usbhid [ 3.203864] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.207601] usbhid: USB HID core driver [ 3.207768] drop_monitor: Initializing network drop monitor service [ 3.207902] Initializing XFRM netlink socket [ 3.208327] NET: Registered protocol family 10 [ 3.217240] Segment Routing with IPv6 [ 3.225206] NET: Registered protocol family 17 [ 3.227088] mpls_gso: MPLS GSO support [ 3.231660] RAS: Correctable Errors collector initialized. [ 3.233403] AVX version of gcm_enc/dec engaged. [ 3.235164] AES CTR mode by8 optimization enabled [ 3.305119] sched_clock: Marking stable (3305019978, 0)->(4174552781, -869532803) [ 3.313848] registered taskstats version 1 [ 3.316877] Loading compiled-in X.509 certificates [ 3.319480] zswap: loaded using pool lzo/zbud [ 3.347910] Key type big_key registered [ 3.363588] Key type encrypted registered [ 3.365450] ima: No TPM chip found, activating TPM-bypass! [ 3.367524] ima: Allocated hash algorithm: sha1 [ 3.369502] ima: No architecture policies found [ 3.371240] evm: Initialising EVM extended attributes: [ 3.373396] evm: security.selinux [ 3.374901] evm: security.ima [ 3.376188] evm: security.capability [ 3.377529] evm: HMAC attrs: 0x1 [ 3.379881] rtc_cmos 00:05: setting system clock to 2026-06-01 14:22:57 UTC (1780323777) [ 3.386499] debug: unmapping init [mem 0xffffffffba603000-0xffffffffba7fffff] [ 3.389381] debug: unmapping init [mem 0xffffffffb9382000-0xffffffffb9658fff] [ 3.398278] Write protecting the kernel read-only data: 28672k [ 3.402315] debug: unmapping init [mem 0xffffffffb7a03000-0xffffffffb7bfffff] [ 3.405460] debug: unmapping init [mem 0xffffffffb8314000-0xffffffffb83fffff] [ 3.438825] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.448571] systemd[1]: Detected virtualization kvm. [ 3.451642] systemd[1]: Detected architecture x86-64. [ 3.453759] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.479582] systemd[1]: No hostname configured. [ 3.481426] systemd[1]: Set hostname to . [ 3.484162] random: systemd: uninitialized urandom read (16 bytes read) [ 3.488012] systemd[1]: Initializing machine ID from random generator. [ 3.647879] random: systemd: uninitialized urandom read (16 bytes read) [ 3.651284] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.657248] random: systemd: uninitialized urandom read (16 bytes read) [ 3.667879] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.687304] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Timers. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.395351] device-mapper: uevent: version 1.0.3 [ 4.397632] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ 4.840734] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.215658] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.336482] scsi host0: ata_piix [ 5.413467] scsi host1: ata_piix [ 5.499707] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.502416] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.911100] random: crng init done [ 9.912893] random: 7 urandom warning(s) missed due to ratelimiting [ 12.731979] dracut-initqueue[592]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 13.424368] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.095373] printk: systemd: 26 output lines suppressed due to ratelimiting [ 15.424263] SELinux: Disabled at runtime. [ 15.495724] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 15.506415] systemd[1]: Detected virtualization kvm. [ 15.508480] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 16.159066] systemd[1]: initrd-switch-root.service: Succeeded. [ 16.163871] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 16.173477] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 16.179973] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 16.185077] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 16.195325] systemd[1]: Starting Journal Service... Starting Journal Service... [ 16.204581] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. Starting Remount Root and Kernel[ 16.540361] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS File Systems... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 17.028322] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.653402] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.707648] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 18.031245] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.054218] EDAC sbridge: Ver: 1.1.2 [ 19.970225] Key type dns_resolver registered [ 20.334704] NFS: Registering the id_resolver key type [ 20.340357] Key type id_resolver registered [ 20.342530] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg311-client login: [ 40.077305] hrtimer: interrupt took 9379785 ns [ 80.508721] libcfs: loading out-of-tree module taints kernel. [ 80.622026] Key type ._llcrypt registered [ 80.623610] Key type .llcrypt registered [ 81.067926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 81.083623] alg: No test for adler32 (adler32-zlib) [ 82.339236] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 83.072235] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 84.760266] Key type lgssc registered [ 86.471633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 175.545866] Lustre: Mounted lustre-client [ 179.018142] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 187.101361] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing check_logdir /tmp/testlogs/ [ 189.969681] Lustre: DEBUG MARKER: oleg311-client.virtnet: executing yml_node [ 192.810137] Lustre: DEBUG MARKER: Client: 2.17.53.29 [ 194.456717] Lustre: DEBUG MARKER: MDS: 2.17.53.29 [ 195.830588] Lustre: DEBUG MARKER: OSS: 2.17.53.29 [ 196.816594] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 1 10:26:10 EDT 2026 [ 201.184442] Lustre: lustre-OST0000-osc-ffff9779043a6000: disconnect after 24s idle [ 206.537886] LustreError: 7921:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9779043a6000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 206.562860] Lustre: Unmounted lustre-client [ 237.176371] Key type lgssc unregistered [ 237.365045] LNet: 8368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 237.370518] LNetError: 8368:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 237.386336] LNet: Removed LNI 192.168.203.11@tcp [ 237.923260] Key type .llcrypt unregistered [ 237.926272] Key type ._llcrypt unregistered [ 254.989915] Key type ._llcrypt registered [ 254.991618] Key type .llcrypt registered [ 255.183828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 255.198190] alg: No test for adler32 (adler32-zlib) [ 256.053565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 256.099777] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 256.111986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 258.871346] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 259.051756] LNet: 9247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.060724] LNetError: 9247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 259.076861] LNet: Removed LNI 192.168.203.11@tcp [ 259.482614] Key type .llcrypt unregistered [ 259.486259] Key type ._llcrypt unregistered [ 263.744356] Key type ._llcrypt registered [ 263.751049] Key type .llcrypt registered [ 264.722273] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:27:17 (1780324037) [ 265.104970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 265.118282] alg: No test for adler32 (adler32-zlib) [ 265.971129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.031925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.054045] LNet: 9657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 266.078666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.120237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 266.138110] LNet: 9664:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 271.185903] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:27:24 (1780324044) [ 271.444306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.472671] LNet: 10241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 271.501514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.595921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 276.164932] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:27:29 (1780324049) [ 276.224813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.244449] LNet: 10822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.260170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.317828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 280.523044] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:27:33 (1780324053) [ 280.570681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.586897] LNet: 11404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.605135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.671063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 284.521206] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:27:37 (1780324057) [ 284.623896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.648590] LNet: 11995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 284.662303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.710918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 284.966994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 289.379391] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:27:42 (1780324062) [ 289.531073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.556699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.613211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 289.899416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 290.187420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 294.365808] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:27:47 (1780324067) [ 294.568112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.589175] LNet: 13180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.592307] LNet: 13180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 294.618726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.689719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 298.968850] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:27:52 (1780324072) [ 299.052218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.083119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.142993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 299.169935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 299.450243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 299.476981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 299.787658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 299.826444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 300.113812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 300.138870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 300.392917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 300.425254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 300.732143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 300.769291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 301.072535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 301.106304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 301.406733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 301.433980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 301.719445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 301.746448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 301.954405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 301.975404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 302.234677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 302.261762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 302.487912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 302.511991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 302.730995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 302.754667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 306.913482] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:28:00 (1780324080) [ 306.962961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.993506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.050638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 307.075279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 311.385264] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:28:04 (1780324084) [ 311.517401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.545987] LNet: 15032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.550198] LNet: 15032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 311.564937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.615511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 311.652249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 315.608558] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:28:08 (1780324088) [ 315.796927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.823920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.876271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 315.908268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 319.934430] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:28:13 (1780324093) [ 320.051187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.109305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.193707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 320.221615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 324.461353] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:28:17 (1780324097) [ 324.534534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 324.566900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.630603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 324.668522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 329.268145] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:28:22 (1780324102) [ 329.339221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.383815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.464640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 329.500516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 333.463440] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:28:26 (1780324106) [ 333.555620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.589442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.690879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 334.035878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 337.864346] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:28:31 (1780324111) [ 338.035254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.059655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.190218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 338.317746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 342.752107] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:28:36 (1780324116) [ 342.890174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.914908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.994832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 343.155740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 347.369921] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:28:40 (1780324120) [ 347.523667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.544278] LNet: 19791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.550367] LNet: 19791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 347.571370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.897688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 348.071314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 352.240598] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:28:45 (1780324125) [ 352.309590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.342610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.410711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 352.446499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 356.609393] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:28:49 (1780324129) [ 356.768318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.801867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.854232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.898831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 361.651561] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:28:54 (1780324134) [ 361.769829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.804311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.864520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 366.361613] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:28:59 (1780324139) [ 366.499885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.538611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.591857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 370.996309] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:29:04 (1780324144) [ 371.072927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.095815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.147929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 371.460721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 375.742123] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:29:08 (1780324148) [ 375.864662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.905214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.962812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 375.988099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 380.773296] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:29:13 (1780324153) [ 380.831204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.855525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.912801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 380.932567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 385.707458] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:29:18 (1780324158) [ 385.780527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.805909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.878480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 385.902735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 390.321884] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:29:23 (1780324163) [ 390.490298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.517910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.570292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 390.599378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 394.547567] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:29:27 (1780324167) [ 394.705814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.740418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.783054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 394.827513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 394.845848] LNetError: 25759:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 394.880744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 394.899096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 398.774564] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:29:32 (1780324172) [ 398.890786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.912628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.966648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jrbLj509cu [ 399.023058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jrbLj509cu [ 399.060736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jrbLj509cu [ 402.696492] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:29:36 (1780324176) [ 402.788582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.821806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.857938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 402.882304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 402.915719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 406.547663] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:29:39 (1780324179) [ 406.664255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.690928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.742530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 408.256137] LNet: Message forwarding will be enabled in 2s [ 411.360104] LNet: Message forwarding enabled [ 411.519918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 411.532683] LNet: Message forwarding will stop in 2s to 6s [ 415.857601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 416.370691] LNet: Message forwarding disabled [ 417.232172] LNet: Message forwarding will be enabled in 2s [ 420.448103] LNet: Message forwarding enabled [ 421.551938] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:29:54 (1780324194) [ 421.622302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.641961] LNet: 28198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.647989] LNet: 28198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 422.426441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.476545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 426.363511] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:29:59 (1780324199) [ 426.426984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.455369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.502829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 426.522355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 426.542653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 426.566541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 426.586853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 426.613993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 426.639319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 426.669884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 426.703511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 426.732592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 426.759838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 426.782861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 426.806383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 426.830234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 426.856749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 426.878552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 426.907546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 426.944912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 426.971860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 426.988579] LNetError: 28826:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 430.787463] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:30:04 (1780324204) [ 430.934313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.964366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.018179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.224095] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:30:08 (1780324208) [ 436.319634] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg311-server [ 437.274480] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:30:10 (1780324210) [ 437.368894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.399023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.428538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 437.441526] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 438.413416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[12-16]@tcp [ 438.427767] LNet: 30541:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 438.448219] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 438.463423] LNetError: 30331:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.12@tcp has gone from up to down [ 439.464458] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 439.478915] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 440.485317] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.12@tcp is being used as a gateway but routing feature is not turned on [ 440.496122] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 441.510586] LNetError: 30336:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.13@tcp has gone from up to down [ 441.520281] LNetError: 30336:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 442.449224] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:30:15 (1780324215) [ 442.537625] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 442.544702] LNetError: 30331:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 443.528615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg311-server [ 444.501609] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:30:17 (1780324217) [ 444.605178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.600193] LNetError: 30330:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.203.12@tcp [ 445.668706] LNetError: 31474:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 445.702087] LNet: Removed LNI 192.168.203.11@tcp [ 445.731413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.808818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 445.830284] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 446.328642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[12-17/2]@tcp [ 446.341603] LNet: 31592:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 446.347405] LNet: 31592:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 446.363306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[12-17/2]@tcp [ 446.373142] LNetError: 31479:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.12@tcp has gone from up to down [ 446.378462] LNetError: 31479:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 450.354278] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:30:23 (1780324223) [ 450.400333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.414745] LNetError: 32179:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 450.424989] LNet: Removed LNI 192.168.203.11@tcp [ 450.438727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.478561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 450.510301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.528347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.566994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.607493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.640342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.689463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 450.746360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.776614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.817917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 450.863779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.887762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.920424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 450.957909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.976434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.015316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.066051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.095216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.133743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 451.178555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.207138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.261729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.317907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.341582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.367082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 451.407252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.432258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.477648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.521749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.547706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.583480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 451.615230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.634537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.679243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.708072] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:30:28 (1780324228) [ 456.789923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg311-server [ 457.705386] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:30:31 (1780324231) [ 458.871987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg311-server [ 459.852300] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:30:33 (1780324233) [ 460.351378] Key type .llcrypt unregistered [ 460.353933] Key type ._llcrypt unregistered [ 463.665338] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 463.791434] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 463.794425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 463.796388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 466.100811] Key type ._llcrypt registered [ 466.108225] Key type .llcrypt registered [ 466.226422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 466.242985] alg: No test for adler32 (adler32-zlib) [ 467.083782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.117967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 467.138493] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 467.521340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 467.533687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 468.715823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 468.730746] LNet: Removed LNI 10.1.2.3@tcp [ 468.816807] LNet: 34302:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.827023] LNetError: 34302:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 468.848247] LNet: Removed LNI 192.168.203.11@tcp [ 469.188496] Key type .llcrypt unregistered [ 469.191492] Key type ._llcrypt unregistered [ 474.880155] Key type ._llcrypt registered [ 474.882523] Key type .llcrypt registered [ 475.669929] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:30:48 (1780324248) [ 476.034539] Key type .llcrypt unregistered [ 476.036337] Key type ._llcrypt unregistered [ 479.171368] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 479.255327] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 479.257706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 479.260263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 480.534044] Key type ._llcrypt registered [ 480.536415] Key type .llcrypt registered [ 480.638252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 480.648708] alg: No test for adler32 (adler32-zlib) [ 481.477218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.514880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 481.530256] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 481.923718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 481.937764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 483.374078] LNet: 35786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.380198] LNetError: 35786:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 483.386110] LNet: Removed LNI 192.168.203.11@tcp [ 483.396222] LNet: Removed LNI 10.1.2.3@tcp [ 483.735932] Key type .llcrypt unregistered [ 483.737768] Key type ._llcrypt unregistered [ 489.281740] Key type ._llcrypt registered [ 489.283978] Key type .llcrypt registered [ 489.983909] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:31:03 (1780324263) [ 490.204666] Key type .llcrypt unregistered [ 490.206526] Key type ._llcrypt unregistered [ 493.353401] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 493.458716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 493.465957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 493.469319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 495.804904] Key type ._llcrypt registered [ 495.806724] Key type .llcrypt registered [ 495.910830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 495.928504] alg: No test for adler32 (adler32-zlib) [ 496.778751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.818905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 496.839883] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 497.864729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 497.878196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 499.003523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 499.015967] LNet: Removed LNI 10.1.2.3@tcp [ 499.026599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 499.050588] LNet: Removed LNI 10.1.2.31@tcp [ 499.062373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 499.083364] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 499.461875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 499.475724] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 499.966877] LNet: 37704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.972331] LNetError: 37704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 499.978066] LNet: Removed LNI 10.1.2.3@tcp [ 499.987539] LNet: Removed LNI 10.1.2.31@tcp [ 500.364963] Key type .llcrypt unregistered [ 500.366724] Key type ._llcrypt unregistered [ 506.311539] Key type ._llcrypt registered [ 506.313789] Key type .llcrypt registered [ 507.081288] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:31:20 (1780324280) [ 507.300225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 507.314523] alg: No test for adler32 (adler32-zlib) [ 508.166376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.216119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 508.236055] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 508.672436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.684622] LNet: 38656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.687955] LNetError: 38656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 508.705127] LNet: Removed LNI 192.168.203.11@tcp [ 508.718662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.749775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.11@tcp [ 508.774603] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 509.172187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.11@tcp [ 509.194825] LNet: Removed LNI 192.168.203.11@tcp [ 509.228620] LNet: 38792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.533444] Key type .llcrypt unregistered [ 509.534838] Key type ._llcrypt unregistered [ 514.500387] Key type ._llcrypt registered [ 514.502250] Key type .llcrypt registered [ 515.145873] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:31:28 (1780324288) [ 516.114491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg311-server [ 517.006748] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:31:30 (1780324290) [ 517.303851] Key type .llcrypt unregistered [ 517.306950] Key type ._llcrypt unregistered [ 520.212698] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 520.289906] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 520.295688] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 520.299109] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 522.664470] Key type ._llcrypt registered [ 522.667548] Key type .llcrypt registered [ 522.788233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 522.798359] alg: No test for adler32 (adler32-zlib) [ 523.663879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.705066] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 524.129084] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 525.367414] LNet: 40625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.373943] LNetError: 40625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 525.378394] LNet: Removed LNI 192.168.203.11@tcp1 [ 525.392910] LNet: Removed LNI 10.1.2.3@tcp2 [ 525.766625] Key type .llcrypt unregistered [ 525.768518] Key type ._llcrypt unregistered [ 531.892943] Key type ._llcrypt registered [ 531.894738] Key type .llcrypt registered [ 532.625830] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:31:45 (1780324305) [ 532.872604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 532.879603] alg: No test for adler32 (adler32-zlib) [ 533.722865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.756964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 533.774888] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 534.343675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.358578] LNet: 41591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.363392] LNetError: 41591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.374541] LNet: Removed LNI 192.168.203.11@tcp [ 534.387063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.421753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 534.434634] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 537.903397] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:31:51 (1780324311) [ 537.966212] LNet: 42177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.973330] LNetError: 42177:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.988387] LNet: Removed LNI 192.168.203.11@tcp [ 538.321560] Key type .llcrypt unregistered [ 538.325399] Key type ._llcrypt unregistered [ 541.134717] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 541.207486] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 541.210291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 541.213524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 543.493485] Key type ._llcrypt registered [ 543.494916] Key type .llcrypt registered [ 543.619777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 543.630773] alg: No test for adler32 (adler32-zlib) [ 544.471351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.508686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 544.524526] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 544.940764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 544.956636] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.033495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.049045] LNet: 42925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 546.054376] LNetError: 42925:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 546.059046] LNet: Removed LNI 192.168.203.11@tcp [ 546.072499] LNet: Removed LNI 10.1.2.3@tcp [ 546.084720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.128043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 546.145911] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 546.151582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.398851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.412847] LNet: Removed LNI 192.168.203.11@tcp [ 546.422345] LNet: Removed LNI 10.1.2.3@tcp [ 546.435539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.476291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 546.492765] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 546.495410] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 546.726921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 546.742406] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 547.378049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.393439] LNet: 43114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.395989] LNet: 43114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 547.399877] LNetError: 43114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.405097] LNetError: 43114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 547.409674] LNet: Removed LNI 192.168.203.11@tcp [ 547.412095] LNet: Removed LNI 10.1.2.3@tcp [ 547.424608] LNet: Removed LNI 192.168.203.11@tcp2 [ 547.438663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.474057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 547.488906] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 547.498965] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 547.503063] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 551.121917] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:32:04 (1780324324) [ 551.192401] LNet: 43703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 551.197131] LNetError: 43703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 551.202578] LNetError: 43703:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 551.208313] LNet: Removed LNI 192.168.203.11@tcp [ 551.211326] LNet: Removed LNI 10.1.2.3@tcp [ 551.221181] LNet: Removed LNI 192.168.203.11@tcp2 [ 551.639306] Key type .llcrypt unregistered [ 551.641155] Key type ._llcrypt unregistered [ 554.700404] Key type ._llcrypt registered [ 554.703669] Key type .llcrypt registered [ 554.821211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 554.834828] alg: No test for adler32 (adler32-zlib) [ 555.689835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.728573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 555.749884] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 556.282449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.299487] LNet: 44105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 556.303757] LNetError: 44105:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.313590] LNet: Removed LNI 192.168.203.11@tcp [ 556.325324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.371406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 556.389690] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 556.653418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.678802] LNet: Removed LNI 192.168.203.11@tcp [ 556.690361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.726989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 556.747332] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 557.014401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.036758] LNet: 44165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 557.045208] LNet: 44165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 557.053619] LNetError: 44165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 557.060864] LNetError: 44165:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 557.085472] LNet: Removed LNI 192.168.203.11@tcp [ 557.097590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.141314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 557.160599] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 557.412197] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 557.463111] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 557.466734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 557.470501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 559.705374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.723787] LNet: 44243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 559.728772] LNetError: 44243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 559.744254] LNet: Removed LNI 192.168.203.11@tcp [ 559.760575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.802254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 559.816485] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 563.833518] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:32:17 (1780324337) [ 563.876562] LNet: 44831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.880782] LNetError: 44831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.893013] LNet: Removed LNI 192.168.203.11@tcp [ 564.303073] Key type .llcrypt unregistered [ 564.305743] Key type ._llcrypt unregistered [ 567.862461] Key type ._llcrypt registered [ 567.863828] Key type .llcrypt registered [ 567.991423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 568.001894] alg: No test for adler32 (adler32-zlib) [ 568.889106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.936642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 568.956077] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 569.513803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.526958] LNet: 45238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.532200] LNetError: 45238:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 569.547421] LNet: Removed LNI 192.168.203.11@tcp [ 569.563627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.614854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 569.632918] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 573.848742] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:32:27 (1780324347) [ 573.965372] LNet: 45830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.969884] LNetError: 45830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.985709] LNet: Removed LNI 192.168.203.11@tcp [ 574.314455] Key type .llcrypt unregistered [ 574.317954] Key type ._llcrypt unregistered [ 577.545770] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 577.642103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 577.645101] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 577.648564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 579.960215] Key type ._llcrypt registered [ 579.961929] Key type .llcrypt registered [ 580.065335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 580.076776] alg: No test for adler32 (adler32-zlib) [ 580.933812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.977519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 581.002083] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 581.568100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.577769] LNet: 46274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.581929] LNetError: 46274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.592981] LNet: Removed LNI 192.168.203.11@tcp [ 581.605807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.646779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 581.663543] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 585.480427] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:32:38 (1780324358) [ 585.526387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.557159] LNet: 46864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.566282] LNetError: 46864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.588158] LNet: Removed LNI 192.168.203.11@tcp [ 585.621103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.685441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 589.034336] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:32:42 (1780324362) [ 589.081672] LNet: 47446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.408827] Key type .llcrypt unregistered [ 589.410916] Key type ._llcrypt unregistered [ 592.093672] Key type ._llcrypt registered [ 592.095128] Key type .llcrypt registered [ 592.180095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 592.192469] alg: No test for adler32 (adler32-zlib) [ 593.014044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.048463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.063626] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 593.502114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.517596] LNet: 47850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 593.521995] LNetError: 47850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 593.538281] LNet: Removed LNI 192.168.203.11@tcp [ 593.552182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.586653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 593.604432] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 596.832710] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:32:50 (1780324370) [ 596.924910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.932595] LNet: 48443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.935087] LNetError: 48443:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.943599] LNet: Removed LNI 192.168.203.11@tcp [ 596.953729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.986582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 600.008605] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:32:53 (1780324373) [ 600.767844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 601.626365] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:32:55 (1780324375) [ 602.449195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 603.394253] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:32:56 (1780324376) [ 604.262645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 605.169303] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:32:58 (1780324378) [ 605.994975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 606.863752] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:33:00 (1780324380) [ 607.717171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 608.683459] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:33:02 (1780324382) [ 608.758333] LNet: 50484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.137925] Key type .llcrypt unregistered [ 609.139626] Key type ._llcrypt unregistered [ 612.082125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 612.175593] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 612.180219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 612.183606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 614.466236] Key type ._llcrypt registered [ 614.471929] Key type .llcrypt registered [ 614.568879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 614.577703] alg: No test for adler32 (adler32-zlib) [ 615.419674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.471128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 615.496992] LNet: Added LNI 192.168.203.11@tcp [8/512/0/180] [ 615.502138] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 615.505330] LNet: 50812:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 615.564375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.580467] LNet: 50831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.584738] LNetError: 50831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.587630] LNet: Removed LNI 192.168.203.11@tcp [ 615.590313] LNetError: 50831:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 615.601239] LNet: Removed LNI 10.1.2.3@tcp [ 615.613195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.652628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 615.675567] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 615.713762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.727267] LNet: 50856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.731665] LNetError: 50856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.743966] LNet: Removed LNI 192.168.203.11@tcp [ 615.757859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.806470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 615.822476] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 615.825775] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 615.828332] LNet: 50864:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 615.880705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 615.893237] LNet: 50882:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.897181] LNetError: 50882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.900873] LNet: Removed LNI 192.168.203.11@tcp [ 615.904339] LNetError: 50882:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 615.924282] LNet: Removed LNI 10.1.2.3@tcp [ 615.937174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.974647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 615.991911] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 615.998045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 616.001067] LNet: 50890:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 616.101594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.122396] LNet: 50916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.128244] LNetError: 50916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.132362] LNet: Removed LNI 192.168.203.11@tcp [ 616.138733] LNetError: 50916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 616.150766] LNet: Removed LNI 10.1.2.3@tcp [ 616.162535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.206529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 616.221463] LNet: Added LNI 192.168.203.11@tcp [8/256/0/182] [ 616.225737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 616.269742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.286740] LNet: 50944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.293279] LNetError: 50944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.297431] LNet: Removed LNI 192.168.203.11@tcp [ 616.299430] LNetError: 50944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 616.311987] LNet: Removed LNI 10.1.2.3@tcp [ 616.325743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.371766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 616.391279] LNet: Added LNI 192.168.203.11@tcp [8/256/0/182] [ 616.394094] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 616.396071] LNet: 50952:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 616.434961] LNet: Removed LNI 192.168.203.11@tcp [ 616.446337] LNet: Removed LNI 10.1.2.3@tcp [ 616.755433] Key type .llcrypt unregistered [ 616.757666] Key type ._llcrypt unregistered [ 622.177718] Key type ._llcrypt registered [ 622.179640] Key type .llcrypt registered [ 622.791489] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:33:16 (1780324396) [ 622.977814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 622.986858] alg: No test for adler32 (adler32-zlib) [ 623.813889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.854972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.867146] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 624.281749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.11 [ 624.294983] LNet: 51927:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 624.301815] LNet: Removed LNI 192.168.203.11@tcp [ 627.499460] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:33:20 (1780324400) [ 627.689260] LNet: 52497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.695641] LNetError: 52497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.708436] LNet: Removed LNI 192.168.203.11@tcp [ 628.070422] Key type .llcrypt unregistered [ 628.072470] Key type ._llcrypt unregistered [ 630.878885] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 630.989638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 633.387357] Key type ._llcrypt registered [ 633.389107] Key type .llcrypt registered [ 633.463529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 633.471052] alg: No test for adler32 (adler32-zlib) [ 634.503966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 637.547663] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:33:31 (1780324411) [ 637.608885] LNet: 53403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.614282] LNetError: 53403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 637.623597] LNet: Removed LNI 10.1.2.3@tcp [ 637.952056] Key type .llcrypt unregistered [ 637.953734] Key type ._llcrypt unregistered [ 640.899543] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 641.026397] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 641.028916] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 641.032413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 643.372764] Key type ._llcrypt registered [ 643.374280] Key type .llcrypt registered [ 643.452899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 643.459517] alg: No test for adler32 (adler32-zlib) [ 644.314142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 647.283808] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:33:40 (1780324420) [ 648.111867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 649.014326] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:33:42 (1780324422) [ 649.132907] LNet: 54591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.137459] LNetError: 54591:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 649.149073] LNet: Removed LNI 10.1.2.3@tcp [ 649.461250] Key type .llcrypt unregistered [ 649.463112] Key type ._llcrypt unregistered [ 652.098366] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 652.225555] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 652.229490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 652.232426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 654.622743] Key type ._llcrypt registered [ 654.625647] Key type .llcrypt registered [ 654.719213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 654.727035] alg: No test for adler32 (adler32-zlib) [ 655.550758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.589066] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 659.803802] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:33:53 (1780324433) [ 660.024151] LNet: 55819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.033094] LNetError: 55819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 660.044603] LNet: Removed LNI 10.1.2.3@tcp [ 660.380598] Key type .llcrypt unregistered [ 660.382947] Key type ._llcrypt unregistered [ 662.881556] Key type ._llcrypt registered [ 662.883272] Key type .llcrypt registered [ 662.997220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.006760] alg: No test for adler32 (adler32-zlib) [ 663.836579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 663.853288] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 667.922643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 668.373345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_interrupt -r 1 [ 668.404756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_interrupt -r 1 [ 668.430257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 668.443361] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 668.558238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 668.579969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 668.605650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 668.625623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 668.647719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 668.668847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 668.689116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 668.712886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 668.730270] LNet: 56256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 668.744489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 668.770967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 668.801553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 668.825989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 668.843226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 668.868449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 668.891338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 669.167899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_dropped -r 1 [ 669.195692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_dropped -r 1 [ 669.215950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 669.227796] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 669.358671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 669.380228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 669.398584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 669.417518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 669.438900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 669.462137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 669.484220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 669.508202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 669.526174] LNet: 56365:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 669.545850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 669.569563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 669.595308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 669.622580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 669.650502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 669.682489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 669.707284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 669.996443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_aborted -r 1 [ 670.022624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_aborted -r 1 [ 670.046243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 670.190704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 670.208390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 670.230254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 670.250802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 670.278175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 670.299538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 670.322344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 670.342802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 670.364310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 670.384313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 670.408105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 670.428402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 670.450255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 670.469950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 670.488537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 670.759912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_no_route -r 1 [ 670.779781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_no_route -r 1 [ 670.802278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 670.813869] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 670.818991] LNet: Skipped 1 previous similar message [ 670.923235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 670.944727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 670.965300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 670.993559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 671.009676] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 671.023227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 671.041747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 671.067581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 671.086473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 671.099574] LNet: 56583:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 671.104786] LNet: 56583:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 671.117506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 671.136916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.164302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 671.188865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 671.219490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 671.247796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 671.273057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 671.526142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_timeout -r 1 [ 671.552839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_timeout -r 1 [ 671.579073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 671.699347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 671.715458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 671.738433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 671.756517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 671.773792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 671.794696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 671.814620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 671.835375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 671.859451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 671.882372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.902208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 671.917356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 671.934879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 671.956022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 671.977966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 672.222307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_error -r 1 [ 672.250758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_error -r 1 [ 672.270574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 672.395681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 672.412162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 672.430683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 672.452717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 672.469400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 672.487061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 672.508975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 672.531871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 672.547836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 672.567925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 672.588315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 672.607903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 672.629529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 672.648548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 672.666676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 672.792389] LNet: 56843:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.797160] LNetError: 56843:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.812338] LNet: Removed LNI 192.168.203.11@tcp [ 673.026515] Key type .llcrypt unregistered [ 673.028834] Key type ._llcrypt unregistered [ 680.640916] Key type ._llcrypt registered [ 680.643099] Key type .llcrypt registered [ 681.350841] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:34:14 (1780324454) [ 681.724253] Key type .llcrypt unregistered [ 681.728250] Key type ._llcrypt unregistered [ 684.625163] Key type ._llcrypt registered [ 684.627131] Key type .llcrypt registered [ 684.745597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 684.753296] alg: No test for adler32 (adler32-zlib) [ 685.619593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 685.635148] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 689.960049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 695.726941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.755825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 695.774427] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 696.692774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_interrupt -r 1 [ 696.720770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 1 [ 696.747269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_interrupt -r 1 [ 696.785168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 1 [ 696.817711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_interrupt -r 1 [ 696.844990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 1 [ 696.873462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_interrupt -r 1 [ 696.900686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 1 [ 696.926951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 696.947729] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 696.952214] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 697.070577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 697.090591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 697.106786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 697.126985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 697.144465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 697.163404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 697.181076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 697.198608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 697.211542] LNet: 58307:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 697.223496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 697.250330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 697.275376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 697.294512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 697.317602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 697.341535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 697.368703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 697.699334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_dropped -r 1 [ 697.721685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 1 [ 697.746709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_dropped -r 1 [ 697.766605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 1 [ 697.796538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_dropped -r 1 [ 697.828169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 1 [ 697.862820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_dropped -r 1 [ 697.889163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 1 [ 697.918959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 697.937266] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 697.944221] LNet: Skipped 2 previous similar messages [ 698.058092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 698.075321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 698.092442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 698.107545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 698.135542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 698.158243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 698.179838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 698.200056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 698.212244] LNet: 58428:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 698.224877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 698.244357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 698.269558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 698.288402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 698.303678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 698.321233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 698.344207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 698.609123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_aborted -r 1 [ 698.631423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 1 [ 698.655237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_aborted -r 1 [ 698.677401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 1 [ 698.701172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_aborted -r 1 [ 698.723716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 1 [ 698.743112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_aborted -r 1 [ 698.767302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 1 [ 698.789257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 698.914378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 698.932940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 698.957557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 698.978325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 698.997174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 699.016783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 699.039258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 699.059392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 699.081847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 699.107621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 699.127103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 699.143427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 699.161601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 699.180413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 699.196118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 699.481320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_no_route -r 1 [ 699.500889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 1 [ 699.522597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_no_route -r 1 [ 699.547916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 1 [ 699.569974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_no_route -r 1 [ 699.598945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 1 [ 699.622557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_no_route -r 1 [ 699.644538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 1 [ 699.676464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 699.692537] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 699.696791] LNet: Skipped 7 previous similar messages [ 699.822493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 699.850225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 699.871352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 699.890695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 699.909673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 699.928872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 699.949144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 699.969048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 699.982354] LNet: 58671:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 699.989189] LNet: 58671:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 700.000554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 700.018610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 700.035108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 700.054428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 700.078865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 700.096797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 700.120146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 700.435933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_timeout -r 1 [ 700.464124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 1 [ 700.489508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_timeout -r 1 [ 700.508108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 1 [ 700.532139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_timeout -r 1 [ 700.581560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 1 [ 700.610104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_timeout -r 1 [ 700.640867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 1 [ 700.666860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 700.780076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 700.797254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 700.813813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 700.829187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 700.846407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 700.875210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 700.891681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 700.914328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 700.938319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 700.954946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 700.978644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 701.000233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 701.019180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 701.036438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 701.058866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 701.311373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_error -r 1 [ 701.337668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_error -r 1 [ 701.362830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_error -r 1 [ 701.384521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_error -r 1 [ 701.413879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_error -r 1 [ 701.433633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_error -r 1 [ 701.461160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_error -r 1 [ 701.484739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_error -r 1 [ 701.506020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 701.617837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 701.637182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 701.659592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 701.679528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 701.703627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 701.730163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 701.752337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 701.771787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 701.788643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 701.806294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 701.832474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 701.851620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 701.873585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 701.895453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 701.920084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 704.910553] LNet: 58989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.914508] LNetError: 58989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.920752] LNet: Removed LNI 192.168.203.11@tcp [ 705.961346] LNet: Removed LNI 192.168.203.11@tcp1 [ 706.200922] Key type .llcrypt unregistered [ 706.203037] Key type ._llcrypt unregistered [ 714.418316] Key type ._llcrypt registered [ 714.419791] Key type .llcrypt registered [ 715.201897] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:34:48 (1780324488) [ 715.744724] Key type .llcrypt unregistered [ 715.748320] Key type ._llcrypt unregistered [ 718.887240] Key type ._llcrypt registered [ 718.889076] Key type .llcrypt registered [ 719.000765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 719.012110] alg: No test for adler32 (adler32-zlib) [ 719.859596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 719.880213] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 724.122061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 724.520482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_dropped -r 1 [ 724.548993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_dropped -r 1 [ 724.576828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 724.592870] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 724.716206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 724.732174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 724.751076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 724.768950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 724.794304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 724.811590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 724.833264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 724.855848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 724.871157] LNet: 60237:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 724.884224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 724.904112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 724.922938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 724.950947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 724.974719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 724.993895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 725.020238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 725.362443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_error -r 1 [ 725.388063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_error -r 1 [ 725.411586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 725.422589] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 725.554843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 725.583739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 725.603689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 725.621868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 725.645405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 725.664947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 725.682747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 725.700369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 725.712352] LNet: 60346:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 725.727367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 725.753220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 725.774208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 725.798276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 725.824934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 725.850506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 725.876648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 726.193979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_timeout -r 1 [ 726.218848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_timeout -r 1 [ 726.239335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 726.369401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 726.389105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 726.415674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 726.439383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 726.461642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 726.488561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 726.517519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 726.538591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 726.563331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 726.586907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 726.616584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 726.642762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 726.671895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 726.706851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 726.729829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 726.895154] LNet: 60496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.903402] LNetError: 60496:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.918912] LNet: Removed LNI 192.168.203.11@tcp [ 727.143749] Key type .llcrypt unregistered [ 727.145930] Key type ._llcrypt unregistered [ 734.458862] Key type ._llcrypt registered [ 734.460218] Key type .llcrypt registered [ 735.168424] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:35:08 (1780324508) [ 735.621393] Key type .llcrypt unregistered [ 735.623215] Key type ._llcrypt unregistered [ 738.573682] Key type ._llcrypt registered [ 738.575817] Key type .llcrypt registered [ 738.687828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.696480] alg: No test for adler32 (adler32-zlib) [ 739.545526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 739.567416] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 743.753326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 747.994535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.023562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 748.038406] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 748.801287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_dropped -r 1 [ 748.827478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 1 [ 748.859647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_dropped -r 1 [ 748.896283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 1 [ 748.925929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_dropped -r 1 [ 748.952073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 1 [ 748.978802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_dropped -r 1 [ 749.007264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 1 [ 749.034720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 749.049915] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 749.054822] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.111@tcp [ 749.198498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 749.220635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 749.249782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 749.275259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 749.307620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 749.334650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 749.365145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 749.385415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 749.405829] LNet: 61960:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 749.423977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 749.448505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 749.471067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 749.501141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 749.522909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.547166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 749.566599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 749.585200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 749.865064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_error -r 1 [ 749.885533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_error -r 1 [ 749.904527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_error -r 1 [ 749.928487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_error -r 1 [ 749.951650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_error -r 1 [ 749.978214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_error -r 1 [ 750.009457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_error -r 1 [ 750.038726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_error -r 1 [ 750.063760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 750.078575] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 750.084844] LNet: Skipped 2 previous similar messages [ 750.197284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 750.225898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 750.252367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 750.274886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 750.301636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 750.322970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 750.341298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 750.357723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 750.368183] LNet: 62084:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 750.381488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 750.398323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 750.418306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 750.434741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 750.455596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.478194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 750.494900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 750.515512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 750.750499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_timeout -r 1 [ 750.770823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_timeout -r 1 [ 750.794845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_timeout -r 1 [ 750.832543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_timeout -r 1 [ 750.857557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_timeout -r 1 [ 750.883763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_timeout -r 1 [ 750.909139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_timeout -r 1 [ 750.934707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_timeout -r 1 [ 750.962940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.111@tcp [ 751.094062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 751.112438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 751.132826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 751.139611] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 751.143262] LNet: Skipped 2 previous similar messages [ 751.157288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 751.174347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 751.192888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 751.212486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 751.229857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 751.247371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 751.265345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 751.287078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 751.305095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 751.325873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 751.342688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 751.360440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 751.387870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 754.551250] LNet: 62284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 754.555855] LNetError: 62284:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 754.562033] LNet: Removed LNI 192.168.203.11@tcp [ 754.570359] LNet: Removed LNI 192.168.203.11@tcp1 [ 754.736509] Key type .llcrypt unregistered [ 754.738315] Key type ._llcrypt unregistered [ 763.130374] Key type ._llcrypt registered [ 763.132812] Key type .llcrypt registered [ 763.894569] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:35:37 (1780324537) [ 764.792342] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 765.722691] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:35:39 (1780324539) [ 766.256638] Key type .llcrypt unregistered [ 766.259317] Key type ._llcrypt unregistered [ 769.459573] Key type ._llcrypt registered [ 769.462329] Key type .llcrypt registered [ 769.594388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.603988] alg: No test for adler32 (adler32-zlib) [ 770.458667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 770.482556] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 774.848862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 775.211917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e network_timeout -r 1 [ 775.234556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e network_timeout -r 1 [ 775.256723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 775.272847] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 775.391383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 775.413469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 775.430551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 775.455722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 775.490822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 775.522368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 775.546098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 775.564753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 775.575737] LNet: 63827:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 775.588693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 775.610995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 775.634482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 775.649092] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 775.659586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 775.681473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 775.703902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 775.831601] LNet: 63866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.837265] LNetError: 63866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.850878] LNet: Removed LNI 192.168.203.11@tcp [ 776.120871] Key type .llcrypt unregistered [ 776.123081] Key type ._llcrypt unregistered [ 784.997881] Key type ._llcrypt registered [ 784.999470] Key type .llcrypt registered [ 785.114259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.123478] alg: No test for adler32 (adler32-zlib) [ 785.967611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 785.984224] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 790.124296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 794.469840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.497078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 794.520645] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 795.388692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e network_timeout -r 1 [ 795.420903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e network_timeout -r 1 [ 795.454871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e network_timeout -r 1 [ 795.486379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e network_timeout -r 1 [ 795.508162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e network_timeout -r 1 [ 795.533449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e network_timeout -r 1 [ 795.557482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e network_timeout -r 1 [ 795.582675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e network_timeout -r 1 [ 795.612806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 795.630821] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 795.792877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 795.820521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 795.840156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 795.867789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 795.895456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 795.918882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 795.944692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 795.965685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 795.982149] LNet: 64743:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 795.999335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 796.032575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 796.059150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 796.085744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 796.106301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 796.127555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 799.082377] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 799.260344] LNet: 64816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.263779] LNetError: 64816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.267611] LNet: Removed LNI 192.168.203.11@tcp [ 799.277851] LNet: Removed LNI 192.168.203.11@tcp1 [ 799.491459] Key type .llcrypt unregistered [ 799.493724] Key type ._llcrypt unregistered [ 807.467802] Key type ._llcrypt registered [ 807.469180] Key type .llcrypt registered [ 808.223883] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:36:21 (1780324581) [ 808.438726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 808.448198] alg: No test for adler32 (adler32-zlib) [ 809.309679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.353994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.376620] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 809.778862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 809.788974] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 810.300857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 810.346275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 810.373579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 810.401178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 810.429741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 810.451000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 810.481239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.11@tcp [ 810.504435] LNetError: 65914:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 811.360227] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 811.360246] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 813.408280] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 813.803302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 817.504331] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 818.185570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 818.207473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 818.243295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 818.260179] LNet: 66032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.263884] LNetError: 66032:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.268994] LNet: Removed LNI 192.168.203.11@tcp [ 818.275625] LNet: Removed LNI 192.168.203.11@tcp1 [ 818.287902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.319385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.336829] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 818.743260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 818.757936] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 819.261160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 819.290749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 819.334383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 819.368968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 819.398627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 819.427566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 819.458611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 819.482748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.11@tcp [ 819.500947] LNetError: 66304:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 820.384469] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 822.758681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 826.528247] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 826.533114] LNet: Skipped 1 previous similar message [ 831.266892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 831.287728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 831.308302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 834.992582] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:36:48 (1780324608) [ 835.113533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.128552] LNet: 67031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.135926] LNetError: 67031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.139581] LNetError: 67031:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 835.144860] LNet: Removed LNI 192.168.203.11@tcp [ 835.155113] LNet: Removed LNI 192.168.203.11@tcp1 [ 835.170081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.214427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 835.229974] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 835.678597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 835.691432] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 836.149331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 836.188696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 836.210835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 836.231811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 836.252247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 836.268669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 836.289669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.11@tcp --health 0 [ 836.339725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 837.216271] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.11@tcp [ 837.216352] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 837.224567] LNet: Skipped 1 previous similar message [ 852.105528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 852.186479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 852.204477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 852.219649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.11@tcp --health 500 [ 852.268493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 859.644503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 859.686129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 859.702682] LNet: 67539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.706879] LNetError: 67539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 859.711459] LNetError: 67539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 859.715477] LNet: Removed LNI 192.168.203.11@tcp [ 859.724420] LNet: Removed LNI 192.168.203.11@tcp1 [ 859.734874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.767357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.783769] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 860.198869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 860.211945] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 860.766714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 860.791190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 860.828540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 860.861885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 860.887537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 860.911210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 860.936819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 860.965758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.11@tcp --health 0 [ 861.793135] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 861.797675] LNet: Skipped 3 previous similar messages [ 862.144754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 871.740186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 871.757274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 871.773804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 875.213521] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:37:28 (1780324648) [ 875.581777] LNet: 68584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.585841] LNetError: 68584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.591330] LNetError: 68584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 875.595924] LNet: Removed LNI 192.168.203.11@tcp [ 875.611068] LNet: Removed LNI 192.168.203.11@tcp1 [ 875.909292] Key type .llcrypt unregistered [ 875.911137] Key type ._llcrypt unregistered [ 878.554975] Key type ._llcrypt registered [ 878.557480] Key type .llcrypt registered [ 878.675314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 878.690356] alg: No test for adler32 (adler32-zlib) [ 879.511805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 879.527058] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 884.405557] Lustre: DEBUG MARKER: Initial discovery [ 884.452391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.111@tcp [ 885.628985] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 885.746874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 885.770456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.111@tcp [ 885.787504] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 885.801305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 885.824687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.111@tcp -r 1 -m GET -l 3 [ 885.843471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.111@tcp -r 1 -m GET -e local_error [ 885.866765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.111@tcp -r 1 -m PUT -l 6 [ 886.595113] Lustre: DEBUG MARKER: Force oleg311-client.virtnet to discover 192.168.203.111@tcp (in background) [ 886.690704] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 886.712811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.111@tcp [ 887.423758] Lustre: DEBUG MARKER: Force oleg311-server to discover 192.168.203.11@tcp [ 888.768322] Lustre: DEBUG MARKER: Wait for 69361 [ 889.760549] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 890.387424] Lustre: DEBUG MARKER: Finished wait on 69361 [ 890.497648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 890.517145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 890.671753] LNet: 69809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.675522] LNetError: 69809:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.687850] LNet: Removed LNI 192.168.203.11@tcp [ 890.882619] Key type .llcrypt unregistered [ 890.885519] Key type ._llcrypt unregistered [ 898.052478] Key type ._llcrypt registered [ 898.054517] Key type .llcrypt registered [ 898.686798] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:37:52 (1780324672) [ 898.920315] Key type .llcrypt unregistered [ 898.922382] Key type ._llcrypt unregistered [ 901.651077] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.714658] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.717601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.720654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.046447] Key type ._llcrypt registered [ 904.047847] Key type .llcrypt registered [ 904.162318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.172083] alg: No test for adler32 (adler32-zlib) [ 905.008113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.037309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 905.059284] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 905.425179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 905.436787] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.424150] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:38:02 (1780324682) [ 909.488614] LNet: 71884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.493612] LNetError: 71884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.499313] LNet: Removed LNI 192.168.203.11@tcp [ 909.505935] LNet: Removed LNI 10.1.2.3@tcp [ 909.820654] Key type .llcrypt unregistered [ 909.822356] Key type ._llcrypt unregistered [ 912.789297] Key type ._llcrypt registered [ 912.791295] Key type .llcrypt registered [ 912.893833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.904734] alg: No test for adler32 (adler32-zlib) [ 913.733607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.763535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.779957] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 914.169754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 914.181687] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 914.704266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.11@tcp [ 915.745153] LNetError: Unexpected error -22 connecting to 192.168.203.11@tcp at host 192.168.203.11:988 [ 917.901800] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:38:11 (1780324691) [ 917.984437] LNet: 73015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.989498] LNetError: 73015:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.993036] LNet: Removed LNI 192.168.203.11@tcp [ 918.003122] LNet: Removed LNI 10.1.2.3@tcp [ 918.324635] Key type .llcrypt unregistered [ 918.327625] Key type ._llcrypt unregistered [ 921.004586] Key type ._llcrypt registered [ 921.007160] Key type .llcrypt registered [ 921.103115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.112115] alg: No test for adler32 (adler32-zlib) [ 921.949552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.982174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 921.993160] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 922.331791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 922.342712] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 922.790404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.11@tcp1 --nid 192.168.203.11@tcp2 [ 924.989615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp1 192.168.203.11@tcp2 [ 925.006130] LNetError: 73852:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.11@tcp1 to 192.168.203.11@tcp2 - Route table may be misconfigured [ 925.021461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp2 192.168.203.11@tcp1 [ 927.837909] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:38:21 (1780324701) [ 927.875387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.888952] LNet: 74419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.896679] LNetError: 74419:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.900962] LNet: Removed LNI 192.168.203.11@tcp1 [ 928.935087] LNet: Removed LNI 192.168.203.11@tcp2 [ 928.950421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.977469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.989664] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 929.325787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 929.338062] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 929.778585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 929.803383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.11@tcp [ 929.823904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp1 192.168.203.11@tcp1 [ 929.848838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.11@tcp -d 192.168.203.11@tcp -e network_timeout [ 929.871221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -e network_timeout [ 929.893715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -e network_timeout [ 929.915451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -e network_timeout [ 929.935836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 929.948101] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 929.958273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 930.004347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 930.017419] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 930.053545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 932.995786] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:38:26 (1780324706) [ 933.039655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.051696] LNet: 75277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.055259] LNetError: 75277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.059574] LNetError: 75277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 933.063739] LNet: Removed LNI 192.168.203.11@tcp [ 934.118686] LNet: Removed LNI 192.168.203.11@tcp1 [ 934.133773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.192719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 934.320849] LNet: 75322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.489566] Key type .llcrypt unregistered [ 934.490920] Key type ._llcrypt unregistered [ 939.472709] Key type ._llcrypt registered [ 939.474526] Key type .llcrypt registered [ 940.114863] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:38:33 (1780324713) [ 940.291788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 940.298583] alg: No test for adler32 (adler32-zlib) [ 941.127038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.864903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 942.779929] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:38:36 (1780324716) [ 942.853425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.870258] LNet: 76417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.892748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.930944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 942.944593] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 943.330969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 943.344792] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 943.813360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 943.830729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp1 [ 943.847946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp1 [ 946.654791] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:38:40 (1780324720) [ 947.540514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg311-server [ 948.270899] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:38:41 (1780324721) [ 949.128698] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg311-server [ 950.008716] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:38:43 (1780324723) [ 950.799370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg311-server [ 951.618331] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:38:45 (1780324725) [ 952.508900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg311-server [ 953.312512] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:38:46 (1780324726) [ 954.201528] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg311-server [ 955.090779] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:38:48 (1780324728) [ 955.990088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg311-server [ 956.744446] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:38:50 (1780324730) [ 957.628613] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg311-server [ 958.386806] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:38:51 (1780324731) [ 959.173369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg311-server [ 959.883530] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:38:53 (1780324733) [ 960.672549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg311-server [ 961.495873] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:38:54 (1780324734) [ 961.539771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.554233] LNet: 80288:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.560599] LNetError: 80288:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.566459] LNet: Removed LNI 192.168.203.11@tcp [ 962.599547] LNet: Removed LNI 192.168.203.11@tcp1 [ 962.613073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.642162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 962.658214] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 963.018118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 963.057551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 964.214171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.232367] LNet: 80448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.237963] LNetError: 80448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.242871] LNetError: 80448:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 964.256542] LNet: Removed LNI 192.168.203.11@tcp [ 964.268248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.298797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 964.312368] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 964.718550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 964.821983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.862741] LNet: Removed LNI 192.168.203.11@tcp [ 965.878927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.910255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 965.922965] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 966.299449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 966.344465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 966.441841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.456160] LNet: 80741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.461627] LNet: 80741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 966.466340] LNetError: 80741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.472706] LNetError: 80741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 966.485055] LNet: Removed LNI 192.168.203.11@tcp [ 966.496686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.537493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 966.555549] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 966.906572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 967.010546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.038443] LNet: Removed LNI 192.168.203.11@tcp [ 968.053959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.085362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 968.101113] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 968.451979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 968.496197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 968.610279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.623716] LNetError: 81033:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.627725] LNetError: 81033:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 968.637940] LNet: Removed LNI 192.168.203.11@tcp [ 968.650964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.696590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 968.719954] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 969.130352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 969.250768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.280837] LNet: Removed LNI 192.168.203.11@tcp [ 970.294147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.323972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 970.339535] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 970.654150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 970.694207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 970.800811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.812301] LNet: 81325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.815991] LNet: 81325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 971.879104] LNet: Removed LNI 192.168.203.11@tcp [ 971.893662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.921387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 971.935784] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 972.308861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 972.417450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 973.479097] LNet: Removed LNI 192.168.203.11@tcp [ 973.490678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 973.531584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 973.555077] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 973.955695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 974.000437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 974.141692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 974.153802] LNetError: 81618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 974.157993] LNetError: 81618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 974.171785] LNet: Removed LNI 192.168.203.11@tcp [ 974.184658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 974.215167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 974.230235] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 974.666913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 974.812650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 975.846837] LNet: Removed LNI 192.168.203.11@tcp [ 975.861565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.910954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.924116] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 976.291714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 976.333208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 977.573559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.599435] LNet: Removed LNI 192.168.203.11@tcp [ 978.615714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.649247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 978.665494] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 979.021449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 980.177082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.187384] LNet: 82073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.191625] LNet: 82073:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 981.229191] LNet: Removed LNI 192.168.203.11@tcp [ 981.240382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.272605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.287498] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 981.627903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 981.658656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.678421] LNet: Removed LNI 192.168.203.11@tcp [ 981.689216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.720989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 981.734126] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 982.091715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 985.622713] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:39:19 (1780324759) [ 985.672425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 985.685484] LNetError: 82908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.689297] LNetError: 82908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 985.703046] LNet: Removed LNI 192.168.203.11@tcp [ 985.713761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.742116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 985.754750] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 986.304524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.325643] LNet: Removed LNI 192.168.203.11@tcp231 [ 986.336890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.368384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 986.383638] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 986.388663] LNet: 83056:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 986.576070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 986.602277] LNet: Removed LNI 192.168.203.11@tcp231 [ 986.611622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 986.623720] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 987.197106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 987.226742] LNet: Removed LNI 192.168.203.11@tcp231 [ 987.241678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 987.293972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 987.311646] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/0] [ 987.314965] LNet: 83208:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 987.572808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 987.591498] LNet: Removed LNI 192.168.203.11@tcp231 [ 987.604938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 987.621629] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/0] [ 988.235761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 988.267820] LNet: Removed LNI 192.168.203.11@tcp231 [ 988.280726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 988.335917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 988.353176] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/60] [ 988.357448] LNet: 83360:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.582419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 988.608694] LNet: Removed LNI 192.168.203.11@tcp231 [ 988.618445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 988.635523] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/60] [ 989.243236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.265870] LNet: Removed LNI 192.168.203.11@tcp231 [ 989.279709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.328640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 989.348076] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 989.566138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 989.589441] LNet: Removed LNI 192.168.203.11@tcp231 [ 989.604856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 989.620351] LNet: Added LNI 192.168.203.11@tcp231 [8/256/0/180] [ 993.423373] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:39:26 (1780324766) [ 993.570365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.590216] LNet: Removed LNI 192.168.203.11@tcp231 [ 993.606681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.655594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 993.675880] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 994.133405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 994.161734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 994.186589] LNet: Removed LNI 192.168.203.11@tcp232 [ 994.197939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 994.213092] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 994.668656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 994.694280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 994.714115] LNet: Removed LNI 192.168.203.11@tcp232 [ 994.722914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 994.738909] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 995.067217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 995.087499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 995.108419] LNet: Removed LNI 192.168.203.11@tcp232 [ 995.116859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 995.130424] LNet: Added LNI 192.168.203.11@tcp232 [8/256/0/180] [ 995.597238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 995.623436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 995.658520] LNet: Removed LNI 192.168.203.11@tcp232 [ 995.673543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 999.209537] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:39:32 (1780324772) [ 999.402895] LNet: 85348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.406264] LNet: 85348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 999.725053] Key type .llcrypt unregistered [ 999.726946] Key type ._llcrypt unregistered [ 1002.860202] Key type ._llcrypt registered [ 1002.861858] Key type .llcrypt registered [ 1002.958902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.966661] alg: No test for adler32 (adler32-zlib) [ 1003.800858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1003.820451] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1007.771798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 1011.498301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.518736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1011.534992] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 1012.251964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_interrupt -r 2 [ 1012.270709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 2 [ 1012.300225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_interrupt -r 2 [ 1012.321689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 2 [ 1012.348654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_interrupt -r 2 [ 1012.368792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_interrupt -r 2 [ 1012.392191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_interrupt -r 2 [ 1012.412178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_interrupt -r 2 [ 1012.433070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 1012.448097] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 1012.452585] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 1012.561328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.577532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1012.597817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.619768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.641857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1012.662249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1012.686088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1012.706639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1012.718591] LNet: 86008:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.731132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1012.747327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1012.767951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1012.785857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1012.805476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.822398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.043323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_dropped -r 2 [ 1013.065106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 2 [ 1013.085647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_dropped -r 2 [ 1013.105959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 2 [ 1013.124838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_dropped -r 2 [ 1013.148456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_dropped -r 2 [ 1013.168826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_dropped -r 2 [ 1013.197492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_dropped -r 2 [ 1013.219836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 1013.230884] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 1013.340387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.355267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1013.371181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1013.385757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1013.406317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.425280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1013.442097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.462818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.476401] LNet: 86128:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1013.488586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1013.506392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1013.526298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1013.547861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1013.564827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1013.588644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.840186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_aborted -r 2 [ 1013.859208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 2 [ 1013.882881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_aborted -r 2 [ 1013.901638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 2 [ 1013.921732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_aborted -r 2 [ 1013.942882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_aborted -r 2 [ 1013.963689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_aborted -r 2 [ 1013.981601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_aborted -r 2 [ 1014.004838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 1014.134159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.151910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1014.173386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.193228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.218615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.237232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.254897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.274399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.293233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.311615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.332995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.355777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.375099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.393828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.664524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_no_route -r 2 [ 1014.686591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 2 [ 1014.709293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_no_route -r 2 [ 1014.739439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 2 [ 1014.775294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_no_route -r 2 [ 1014.811267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_no_route -r 2 [ 1014.842965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_no_route -r 2 [ 1014.869281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_no_route -r 2 [ 1014.896965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 1014.910698] LNet: There was an unexpected network error while writing to 192.168.203.111: rc = -22 [ 1014.916460] LNet: Skipped 2 previous similar messages [ 1015.049380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.069876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.090753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.112852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.141901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.163741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.181673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.199905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.209976] LNet: 86368:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.214889] LNet: 86368:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1015.230663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.259400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.282138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.305966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.326878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.345102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.613934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e local_timeout -r 2 [ 1015.635597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 2 [ 1015.654712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e local_timeout -r 2 [ 1015.677903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 2 [ 1015.704861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e local_timeout -r 2 [ 1015.735580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e local_timeout -r 2 [ 1015.758235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e local_timeout -r 2 [ 1015.782940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e local_timeout -r 2 [ 1015.811102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 1015.952940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.972518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1015.989864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.007875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.045554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.073870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.092791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.118298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.146697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.164944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.183825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.199703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.220292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.251088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.500427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp -m GET -e remote_dropped -r 2 [ 1016.525113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 2 [ 1016.545730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp -m GET -e remote_dropped -r 2 [ 1016.568512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.11@tcp -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 2 [ 1016.590283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp -m GET -e remote_dropped -r 2 [ 1016.615413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.111@tcp1 -m GET -e remote_dropped -r 2 [ 1016.641377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp -m GET -e remote_dropped -r 2 [ 1016.666124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.11@tcp1 -d 192.168.203.11@tcp1 -m GET -e remote_dropped -r 2 [ 1016.697347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.11@tcp 192.168.203.111@tcp [ 1016.716406] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.111@tcp [ 1016.837731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.861116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1016.883411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.908251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.926982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.945819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.973647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.000885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.023909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.043528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.070332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.096708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.118191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.139321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.887193] LNet: 86682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.890530] LNetError: 86682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.895089] LNet: Removed LNI 192.168.203.11@tcp [ 1019.903518] LNet: Removed LNI 192.168.203.11@tcp1 [ 1020.079617] Key type .llcrypt unregistered [ 1020.081133] Key type ._llcrypt unregistered [ 1027.312332] Key type ._llcrypt registered [ 1027.314273] Key type .llcrypt registered [ 1027.933789] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:40:01 (1780324801) [ 1028.122048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1028.129164] alg: No test for adler32 (adler32-zlib) [ 1028.950130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.982163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1028.997693] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1029.403917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1029.425651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.11@tcp [ 1029.441875] LNetError: 87658:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 1029.491118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.11@tcp [ 1029.544689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.402420] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:40:05 (1780324805) [ 1032.451735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1032.472578] LNet: 88242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.478191] LNetError: 88242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1032.493426] LNet: Removed LNI 192.168.203.11@tcp [ 1032.506111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1032.534120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1032.550444] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1032.928618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1032.938564] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 1033.386242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.11@tcp [ 1033.399783] LNetError: 88513:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.11@tcp to recovery [ 1033.568339] LNet: 1 local NIs in recovery (showing 1): 192.168.203.11@tcp [ 1033.570328] LNet: There was an unexpected network error while writing to 192.168.203.11: rc = -22 [ 1034.503535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.11@tcp [ 1039.528670] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:40:13 (1780324813) [ 1040.526189] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg311-server [ 1041.294390] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:40:14 (1780324814) [ 1042.098677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg311-server [ 1042.925587] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:40:16 (1780324816) [ 1043.744664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg311-server [ 1044.535562] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:40:17 (1780324817) [ 1044.580591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1044.594311] LNet: 90137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1044.598540] LNetError: 90137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1044.601768] LNet: Removed LNI 192.168.203.11@tcp [ 1044.607590] LNet: Removed LNI 192.168.203.11@tcp1 [ 1044.618285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1044.652922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1044.669996] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1045.066695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1045.103493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1045.123377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1045.160378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1045.185603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1045.212292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1048.160892] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:40:21 (1780324821) [ 1048.273238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1048.290446] LNet: 90868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.296152] LNetError: 90868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.299978] LNetError: 90868:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1048.310841] LNet: Removed LNI 192.168.203.11@tcp [ 1048.324142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.361271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1048.377735] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1051.872900] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:40:25 (1780324825) [ 1051.970547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1051.983304] LNet: 91570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.986345] LNetError: 91570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1051.994328] LNet: Removed LNI 192.168.203.11@tcp [ 1052.005113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.032972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1052.044956] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1055.259913] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:40:28 (1780324828) [ 1056.025552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1056.908321] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:40:30 (1780324830) [ 1057.162226] LNet: 92604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.167861] LNetError: 92604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.179769] LNet: Removed LNI 192.168.203.11@tcp [ 1057.513738] Key type .llcrypt unregistered [ 1057.516171] Key type ._llcrypt unregistered [ 1060.212973] Key type ._llcrypt registered [ 1060.214359] Key type .llcrypt registered [ 1060.310329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.320945] alg: No test for adler32 (adler32-zlib) [ 1061.159468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1061.173605] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1065.116912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 1067.206447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.111@tcp [ 1067.386110] LNet: 93020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.391268] LNetError: 93020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.405039] LNet: Removed LNI 192.168.203.11@tcp [ 1067.629617] Key type .llcrypt unregistered [ 1067.631445] Key type ._llcrypt unregistered [ 1073.282628] Key type ._llcrypt registered [ 1073.284268] Key type .llcrypt registered [ 1074.000720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1074.885573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1075.803335] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:40:49 (1780324849) [ 1076.575313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1077.433795] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:40:50 (1780324850) [ 1078.379229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg311-server [ 1079.158840] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:40:52 (1780324852) [ 1080.048906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg311-server [ 1080.842690] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:40:54 (1780324854) [ 1081.366322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1081.411021] alg: No test for adler32 (adler32-zlib) [ 1082.321265] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1082.445497] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1084.056520] Key type lgssc registered [ 1084.599658] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1094.212617] Key type lgssc unregistered [ 1094.341398] LNet: 96194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1094.344369] LNetError: 96194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1094.359572] LNet: Removed LNI 192.168.203.11@tcp [ 1094.684422] Key type .llcrypt unregistered [ 1094.685758] Key type ._llcrypt unregistered [ 1101.180053] Key type ._llcrypt registered [ 1101.181445] Key type .llcrypt registered [ 1101.800315] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:41:15 (1780324875) [ 1101.969712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1101.977919] alg: No test for adler32 (adler32-zlib) [ 1102.802354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1102.834181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1102.855991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1102.882776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1102.905686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1105.801549] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:41:19 (1780324879) [ 1105.898179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1105.909156] LNet: 97606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1105.920826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.958102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1105.974367] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1106.398453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.410705] LNet: 97750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.414406] LNetError: 97750:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.424333] LNet: Removed LNI 192.168.203.11@tcp [ 1106.433548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.460315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1106.477440] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1106.699764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.722657] LNet: Removed LNI 192.168.203.11@tcp [ 1106.731905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1106.769436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1106.787404] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1109.745737] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:41:23 (1780324883) [ 1109.797722] LNet: 98363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.805768] LNet: 98363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1109.810104] LNetError: 98363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1109.813230] LNetError: 98363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1109.820181] LNet: Removed LNI 192.168.203.11@tcp [ 1110.116538] Key type .llcrypt unregistered [ 1110.118777] Key type ._llcrypt unregistered [ 1112.649622] Key type ._llcrypt registered [ 1112.651258] Key type .llcrypt registered [ 1112.740247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.747631] alg: No test for adler32 (adler32-zlib) [ 1113.561558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1113.601495] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1113.605272] LNet: *** cfs_fail_loc=e003, val=100*** [ 1113.945163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 1113.955722] Ping reply received with mapping: [ 1113.957838] found metadata reply NI[192.168.203.11@tcp] [ 1113.960017] 0b cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1113.961874] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1113.963976] 5a 5a 5a 5a 5a 5a 5a 5a [ 1114.020476] LNet: 98765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.023248] LNetError: 98765:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.031520] LNet: Removed LNI 192.168.203.11@tcp [ 1114.323484] Key type .llcrypt unregistered [ 1114.325821] Key type ._llcrypt unregistered [ 1118.979557] Key type ._llcrypt registered [ 1118.981164] Key type .llcrypt registered [ 1119.592341] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:41:33 (1780324893) [ 1119.822550] Key type .llcrypt unregistered [ 1119.824608] Key type ._llcrypt unregistered [ 1122.290888] Key type ._llcrypt registered [ 1122.292994] Key type .llcrypt registered [ 1122.405556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1122.415048] alg: No test for adler32 (adler32-zlib) [ 1123.350220] LNetError: 99810:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1123.545803] Key type .llcrypt unregistered [ 1123.547524] Key type ._llcrypt unregistered [ 1125.878792] Key type ._llcrypt registered [ 1125.879983] Key type .llcrypt registered [ 1126.473796] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:41:39 (1780324899) [ 1126.692360] Key type .llcrypt unregistered [ 1126.694392] Key type ._llcrypt unregistered [ 1129.121069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1129.124931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1131.339524] Key type ._llcrypt registered [ 1131.340883] Key type .llcrypt registered [ 1131.414895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1131.421665] alg: No test for adler32 (adler32-zlib) [ 1132.236610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.269738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1136.994107] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:41:50 (1780324910) [ 1137.105387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1137.120577] LNet: 101589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1137.124919] LNetError: 101589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1137.134186] LNet: Removed LNI 10.1.2.3@tcp [ 1137.145889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1137.464467] Key type .llcrypt unregistered [ 1137.466371] Key type ._llcrypt unregistered [ 1139.874884] Key type ._llcrypt registered [ 1139.876594] Key type .llcrypt registered [ 1139.952655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.959321] alg: No test for adler32 (adler32-zlib) [ 1140.781233] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1140.809837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 1141.923444] LNet: 101886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.932068] LNetError: 101886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1141.944749] LNet: Removed LNI 192.168.203.11@tcp [ 1142.216963] Key type .llcrypt unregistered [ 1142.219083] Key type ._llcrypt unregistered [ 1146.553746] Key type ._llcrypt registered [ 1146.555319] Key type .llcrypt registered [ 1147.112106] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:42:00 (1780324920) [ 1147.405441] Key type .llcrypt unregistered [ 1147.407394] Key type ._llcrypt unregistered [ 1149.780207] Key type ._llcrypt registered [ 1149.781549] Key type .llcrypt registered [ 1149.869941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1149.876633] alg: No test for adler32 (adler32-zlib) [ 1153.610581] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:42:07 (1780324927) [ 1153.689294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1153.718903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1153.753523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1153.770975] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1154.091965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1154.108891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1154.130928] LNet: Removed LNI 192.168.203.11@tcp [ 1154.234140] LNet: 103629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1154.416609] Key type .llcrypt unregistered [ 1154.420279] Key type ._llcrypt unregistered [ 1158.852742] Key type ._llcrypt registered [ 1158.854589] Key type .llcrypt registered [ 1159.428136] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:42:12 (1780324932) [ 1159.595629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.605302] alg: No test for adler32 (adler32-zlib) [ 1160.419948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1160.447755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1160.459523] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1160.786591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 1160.806681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.11@tcp [ 1160.816475] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1160.818852] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1160.820928] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1160, 0, 0 [ 1160.824056] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1160.825857] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1160.827889] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1160.829864] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1160, 0, 0 [ 1160.832809] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1160.834243] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1160.836098] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1160.837807] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1160, 0, 1210 [ 1160.841063] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1160.843088] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1160.846725] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1160.849273] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1160, 0, 1210 [ 1160.852871] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1163.732646] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:42:17 (1780324937) [ 1163.927848] LNet: 105166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.932450] LNetError: 105166:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.950913] LNet: Removed LNI 192.168.203.11@tcp [ 1164.258296] Key type .llcrypt unregistered [ 1164.260347] Key type ._llcrypt unregistered [ 1167.174655] Key type ._llcrypt registered [ 1167.176616] Key type .llcrypt registered [ 1167.323752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1167.333424] alg: No test for adler32 (adler32-zlib) [ 1168.157982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1168.174785] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1171.899094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 1176.168115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1176.189591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1176.203909] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 1176.647966] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1176.702255] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1176.704932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1176.707766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1178.894744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1178.908185] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1183.630970] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:42:37 (1780324957) [ 1183.728538] LNet: 106670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.732434] LNetError: 106670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.736055] LNet: Removed LNI 192.168.203.11@tcp [ 1184.800156] LNet: Removed LNI 192.168.203.11@tcp1 [ 1184.802912] LNetError: 106670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1184.806592] LNetError: 106670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1184.817159] LNet: Removed LNI 10.1.2.3@tcp99 [ 1185.102302] Key type .llcrypt unregistered [ 1185.106822] Key type ._llcrypt unregistered [ 1187.790023] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1187.796515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1190.065726] Key type ._llcrypt registered [ 1190.068684] Key type .llcrypt registered [ 1190.172102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1190.181306] alg: No test for adler32 (adler32-zlib) [ 1191.005524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.035327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1191.052337] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1191.422626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1191.435403] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1192.394820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1192.416446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1192.434948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1192.446347] LNetError: 107414:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1192.460937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1192.482133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1192.664545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1192.695817] LNet: 107434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.699242] LNetError: 107434:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.703111] LNet: Removed LNI 192.168.203.11@tcp [ 1192.710398] LNet: Removed LNI 10.1.2.3@tcp [ 1193.033536] Key type .llcrypt unregistered [ 1193.035169] Key type ._llcrypt unregistered [ 1197.565508] Key type ._llcrypt registered [ 1197.568690] Key type .llcrypt registered [ 1198.273356] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:42:51 (1780324971) [ 1198.448214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.457916] alg: No test for adler32 (adler32-zlib) [ 1199.285790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.315417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1199.329191] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1199.678675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.11@tcp [ 1199.717728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg311-client@tcp [ 1202.475451] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:42:55 (1780324975) [ 1202.545826] LNet: 108961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.548488] LNetError: 108961:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1202.557179] LNet: Removed LNI 192.168.203.11@tcp [ 1202.872343] Key type .llcrypt unregistered [ 1202.874188] Key type ._llcrypt unregistered [ 1205.428349] Key type ._llcrypt registered [ 1205.430900] Key type .llcrypt registered [ 1205.530230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1205.537835] alg: No test for adler32 (adler32-zlib) [ 1206.359754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1206.375325] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1206.394230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1209.434305] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:43:02 (1780324982) [ 1210.136425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1210.913394] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:43:04 (1780324984) [ 1211.599561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1212.459613] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:43:05 (1780324985) [ 1212.530667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1212.547867] LNet: 110382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.552184] LNetError: 110382:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.561850] LNet: Removed LNI 192.168.203.11@tcp [ 1212.571143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.601648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1212.616467] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1213.031038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1213.064565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1216.323931] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:43:09 (1780324989) [ 1216.390815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1216.408993] LNet: 111090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.414958] LNetError: 111090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.422955] LNet: Removed LNI 192.168.203.11@tcp [ 1216.435824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1216.466992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1216.494585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1216.783623] Key type .llcrypt unregistered [ 1216.785268] Key type ._llcrypt unregistered [ 1221.479920] Key type ._llcrypt registered [ 1221.481481] Key type .llcrypt registered [ 1222.129432] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:43:15 (1780324995) [ 1222.313487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1222.321529] alg: No test for adler32 (adler32-zlib) [ 1223.150340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1223.181610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1223.195760] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1223.550504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1223.582674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1226.659139] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:43:20 (1780325000) [ 1226.701960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.714646] LNet: 112639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.719068] LNetError: 112639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.752132] LNet: Removed LNI 192.168.203.11@tcp [ 1227.763962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.796157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1227.812778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1230.883963] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:43:24 (1780325004) [ 1231.008461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1231.021588] LNet: 113225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.032603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.068763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1231.264891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1231.287842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1231.323446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1231.498155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1231.523678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1231.573903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1231.602748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.640578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1231.817781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1231.835323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1231.866811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1232.040197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1232.060515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1235.106455] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:43:28 (1780325008) [ 1235.235080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1235.249799] LNet: 113858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1235.253599] LNet: 113858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1235.265312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.296934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1235.309827] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1235.665434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1235.676709] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1236.108942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1236.136733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1236.236345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1236.246930] LNetError: 114137:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.249898] LNet: Removed LNI 192.168.203.11@tcp [ 1236.256648] LNet: Removed LNI 192.168.203.11@tcp2 [ 1236.266982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.293159] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1236.296579] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1236.301216] LNet: 114143:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.519481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1236.620532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.664307] LNet: Removed LNI 192.168.203.11@tcp [ 1237.666650] LNetError: 114180:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.674438] LNetError: 114180:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1237.692576] LNet: Removed LNI 192.168.203.11@tcp2 [ 1237.706173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.736526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1237.748788] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1238.123001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1238.137487] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1238.539571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1238.559604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1238.593384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1238.706397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1238.725174] LNet: 114465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1238.730484] LNet: 114465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1239.776121] LNet: Removed LNI 192.168.203.11@tcp [ 1239.778932] LNetError: 114465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.784175] LNetError: 114465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1239.794831] LNet: Removed LNI 192.168.203.11@tcp2 [ 1239.810044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.847813] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1239.852038] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1239.855957] LNet: 114471:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.084112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1240.187079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1240.200367] LNet: Removed LNI 192.168.203.11@tcp [ 1241.256201] LNet: Removed LNI 192.168.203.11@tcp2 [ 1241.270260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.301340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1241.316933] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1241.670812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1241.680302] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1242.131442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1242.167385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1242.287356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.360172] LNet: Removed LNI 192.168.203.11@tcp [ 1243.369628] LNet: Removed LNI 192.168.203.11@tcp2 [ 1243.388978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.426093] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1243.429712] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1243.433373] LNet: 114793:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.661166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1243.773764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1243.788235] LNet: 114830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.792639] LNet: 114830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1243.800298] LNetError: 114830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.803380] LNetError: 114830:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1243.807872] LNet: Removed LNI 192.168.203.11@tcp [ 1243.821953] LNet: Removed LNI 192.168.203.11@tcp2 [ 1243.834977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.867075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1243.878563] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1244.237880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1244.251913] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1244.716795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1244.738389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1244.767679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1244.887387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1245.920140] LNet: Removed LNI 192.168.203.11@tcp [ 1246.950759] LNet: Removed LNI 192.168.203.11@tcp2 [ 1246.972971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.022215] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1247.026549] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1247.029905] LNet: 115121:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1247.307907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1247.405702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1247.420209] LNet: Removed LNI 192.168.203.11@tcp [ 1248.487136] LNet: Removed LNI 192.168.203.11@tcp2 [ 1248.497741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1248.526467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1248.541345] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1248.939128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1248.954427] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1249.410339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1249.432057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1249.463512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1249.644487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1250.720852] LNet: Removed LNI 192.168.203.11@tcp [ 1250.737526] LNet: Removed LNI 192.168.203.11@tcp2 [ 1250.754258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1250.795814] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1250.799783] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1251.028628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1251.181975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1252.256139] LNet: Removed LNI 192.168.203.11@tcp [ 1252.258795] LNetError: 115512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.263823] LNetError: 115512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1252.275483] LNet: Removed LNI 192.168.203.11@tcp2 [ 1252.290255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.326735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1252.342844] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1252.681851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1252.694579] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1253.121860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1253.150275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1253.169215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1253.200917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1253.387914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.401706] LNet: 115814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.406312] LNet: 115814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1254.432239] LNet: Removed LNI 192.168.203.11@tcp [ 1255.462506] LNet: Removed LNI 192.168.203.11@tcp2 [ 1255.475174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.513589] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1255.518970] LNet: Added LNI 192.168.203.11@tcp2 [8/256/0/180] [ 1255.521925] LNet: 115820:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1255.525237] LNet: 115820:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1255.744768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1258.997649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1259.676444] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:43:53 (1780325033) [ 1259.739695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1259.752271] LNet: Removed LNI 192.168.203.11@tcp [ 1260.774685] LNet: Removed LNI 192.168.203.11@tcp2 [ 1260.786532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1260.923742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1260.937558] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1261.257069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1261.269302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1261.649528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1261.660440] LNet: Removed LNI 192.168.203.11@tcp [ 1261.667446] LNet: Removed LNI 10.1.2.3@tcp [ 1261.678445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1261.713518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1261.725473] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1261.729189] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1261.899345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1261.913359] LNet: Removed LNI 192.168.203.11@tcp [ 1261.920600] LNet: Removed LNI 10.1.2.3@tcp [ 1264.521302] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:43:58 (1780325038) [ 1264.571263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1264.595274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.644876] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1264.708960] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1264.718294] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1264.721820] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1266.853263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1266.864385] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1267.197534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1267.212804] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1268.089489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.11@tcp [ 1268.109684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1268.141402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.11@tcp -m PUT -e local_timeout -r 1 [ 1268.171690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1268.202197] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1268.205932] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1268.306818] LNetError: 117517:0:(lib-move.c:4710:lnet_parse()) 192.168.203.11@tcp, src 192.168.203.11@tcp: Dropping PUT (error -108 looking up sender) [ 1269.345334] LNetError: 117975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.349733] LNetError: 117975:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1269.354052] LNet: Removed LNI 192.168.203.11@tcp [ 1269.364356] LNet: Removed LNI 10.1.2.3@tcp [ 1269.553407] Key type .llcrypt unregistered [ 1269.555307] Key type ._llcrypt unregistered [ 1274.014741] Key type ._llcrypt registered [ 1274.016375] Key type .llcrypt registered [ 1274.599344] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:44:08 (1780325048) [ 1274.751197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1274.757700] alg: No test for adler32 (adler32-zlib) [ 1275.575476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1275.738220] LNet: 118826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.995308] Key type .llcrypt unregistered [ 1275.996910] Key type ._llcrypt unregistered [ 1278.280459] Key type ._llcrypt registered [ 1278.281696] Key type .llcrypt registered [ 1278.367847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.378478] alg: No test for adler32 (adler32-zlib) [ 1279.198285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1279.212389] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1282.541633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 1282.618271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1282.880366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.111@tcp [ 1282.985518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.006572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1283.023532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.039532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.057759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.077667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.097661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.113825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.122131] LNet: 119275:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.131163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.152050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.169944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.190764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.211389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.233323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.521101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1283.647903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.664138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1283.680079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.695753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.712463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.726654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.744481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.759876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.769795] LNet: 119392:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.779644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.797437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.816324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.832287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.853378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.879970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.149877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.111@tcp [ 1284.248450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.264648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1284.279791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.295462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.310832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.320526] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.111@tcp [ 1284.332296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.352608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.371930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1284.393132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1284.408939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1284.431036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1284.450242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.465504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.481552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.752361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1284.860982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1284.877430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1284.899061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1284.915549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1284.936792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1284.954135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1284.970102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1284.990564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.002845] LNet: 119626:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1285.005976] LNet: 119626:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1285.016906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.036604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.055347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.078722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.105375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.125428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.262075] LNet: 119672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.265365] LNetError: 119672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.276489] LNet: Removed LNI 192.168.203.11@tcp [ 1285.452097] Key type .llcrypt unregistered [ 1285.453521] Key type ._llcrypt unregistered [ 1291.730195] Key type ._llcrypt registered [ 1291.731635] Key type .llcrypt registered [ 1292.310864] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:44:25 (1780325065) [ 1292.483453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1292.490574] alg: No test for adler32 (adler32-zlib) [ 1293.307430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.469380] LNet: 120557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.741317] Key type .llcrypt unregistered [ 1293.742828] Key type ._llcrypt unregistered [ 1296.484663] Key type ._llcrypt registered [ 1296.487186] Key type .llcrypt registered [ 1296.592951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.599782] alg: No test for adler32 (adler32-zlib) [ 1297.416564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1297.431650] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1300.883619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.111@tcp [ 1304.148854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.168263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1304.180476] LNet: Added LNI 192.168.203.11@tcp1 [8/256/0/180] [ 1304.578521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1304.823733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.111@tcp [ 1304.932883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.950677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.970211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.988164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.009594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.027310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.045469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.062638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.072806] LNet: 121207:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.083141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.101409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.119263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.136101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.152540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.168954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1305.486184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1305.585269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.601585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1305.619345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.632201] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.111@tcp, 192.168.203.111@tcp1 [ 1305.635505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.652468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.666795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.685924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.703122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.713565] LNet: 121324:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.725155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.744128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.759641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.774942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1305.792402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1305.809813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.061871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.111@tcp [ 1306.160091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.175428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.192081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.209364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.223346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.237591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.256487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.279169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.300570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.323422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.343928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.363702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.382167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.399097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.697863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1306.824472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1306.846966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1306.864741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.882794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.902928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.923251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.942242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.961512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.972585] LNet: 121558:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.976411] LNet: 121558:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1306.988836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.005628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.021554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1307.038626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.055933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1307.073972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.472221] LNet: 121639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.475606] LNetError: 121639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.479508] LNet: Removed LNI 192.168.203.11@tcp [ 1309.488506] LNet: Removed LNI 192.168.203.11@tcp1 [ 1309.653650] Key type .llcrypt unregistered [ 1309.655444] Key type ._llcrypt unregistered [ 1315.726869] Key type ._llcrypt registered [ 1315.728677] Key type .llcrypt registered [ 1316.290840] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:44:49 (1780325089) [ 1319.098586] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:44:52 (1780325092) [ 1321.893472] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:44:55 (1780325095) [ 1324.876437] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:44:58 (1780325098) [ 1327.887492] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:45:01 (1780325101) [ 1330.968164] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:45:04 (1780325104) [ 1333.901803] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:45:07 (1780325107) [ 1337.024791] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:45:10 (1780325110) [ 1340.360679] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:45:13 (1780325113) [ 1343.408846] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:45:16 (1780325116) [ 1346.188868] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:45:19 (1780325119) [ 1349.221172] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:45:22 (1780325122) [ 1352.105508] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:45:25 (1780325125) [ 1355.058248] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:45:28 (1780325128) [ 1358.053967] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:45:31 (1780325131) [ 1361.176526] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:45:34 (1780325134) [ 1364.074997] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:45:37 (1780325137) [ 1366.996117] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:45:40 (1780325140) [ 1367.186380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1367.193094] alg: No test for adler32 (adler32-zlib) [ 1368.002757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.058898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1368.073500] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1368.538311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.547771] LNet: 132335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1368.552138] LNetError: 132335:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1368.561346] LNet: Removed LNI 192.168.203.11@tcp [ 1368.570645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.596996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1368.608875] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1371.262690] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:45:44 (1780325144) [ 1374.131429] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:45:47 (1780325147) [ 1377.024175] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:45:50 (1780325150) [ 1379.841034] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:45:53 (1780325153) [ 1382.501798] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:45:56 (1780325156) [ 1385.275343] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:45:58 (1780325158) [ 1388.067990] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:46:01 (1780325161) [ 1390.691159] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:46:04 (1780325164) [ 1393.188232] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:46:06 (1780325166) [ 1395.895959] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:46:09 (1780325169) [ 1398.411869] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:46:12 (1780325172) [ 1400.811353] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:46:14 (1780325174) [ 1403.225151] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1205 sec ========== 10:46:16 (1780325176) [ 1403.320341] LNet: 139791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.324132] LNetError: 139791:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.331872] LNet: Removed LNI 192.168.203.11@tcp [ 1403.595749] Key type .llcrypt unregistered [ 1403.597171] Key type ._llcrypt unregistered [ 1408.008215] Key type ._llcrypt registered [ 1408.009938] Key type .llcrypt registered [ 1408.236489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1408.244489] alg: No test for adler32 (adler32-zlib) [ 1409.129326] Lustre: Lustre: Build Version: 2.17.53_29_g1eaec4e [ 1409.238418] LNet: Added LNI 192.168.203.11@tcp [8/256/0/180] [ 1410.832142] Key type lgssc registered [ 1411.315246] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1436.249761] Lustre: Mounted lustre-client [ 1438.923600] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1441.484187] LustreError: 142227:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff977920c60000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1441.508158] Lustre: Unmounted lustre-client [ 1453.755191] Key type lgssc unregistered [ 1453.883355] LNet: 142706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.887095] LNetError: 142706:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.894793] LNet: Removed LNI 192.168.203.11@tcp [ 1454.150124] Key type .llcrypt unregistered [ 1454.151287] Key type ._llcrypt unregistered