[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 413310623 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003145] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010099] LSM: Security Framework initializing [ 0.011050] Yama: becoming mindful. [ 0.012043] SELinux: Initializing. [ 0.014012] *** VALIDATE selinux *** [ 0.022582] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027467] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.032267] *** VALIDATE proc *** [ 0.033233] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036123] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.043006] debug: unmapping init [mem 0xffffffffa3c59000-0xffffffffa3c60fff] [ 0.045175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047024] ... version: 2 [ 0.048010] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054272] rcu: Hierarchical SRCU implementation. [ 0.056465] smp: Bringing up secondary CPUs ... [ 0.057597] x86: Booting SMP configuration: [ 0.058024] .... node #0, CPUs: #1 #2 #3 [ 0.061253] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141504] node 0 deferred pages initialised in 75ms [ 0.144125] devtmpfs: initialized [ 0.145221] x86/mm: Memory block size: 128MB [ 0.147813] gcov: version magic: 0x41383552 [ 0.149117] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.150074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.151208] pinctrl core: initialized pinctrl subsystem [ 0.152197] [ 0.152758] ************************************************************* [ 0.153014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.154016] ** ** [ 0.155013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156011] ** ** [ 0.157010] ** This means that this kernel is built to expose internal ** [ 0.158015] ** IOMMU data structures, which may compromise security on ** [ 0.159014] ** your system. ** [ 0.160014] ** ** [ 0.161016] ** If you see this message and you are not debugging the ** [ 0.162013] ** kernel, report this immediately to your vendor! ** [ 0.163012] ** ** [ 0.164013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165013] ************************************************************* [ 0.166619] NET: Registered protocol family 16 [ 0.167457] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.169058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.171013] cpuidle: using governor menu [ 0.172696] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.175524] PCI: Using configuration type 1 for base access [ 0.178113] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.188030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192030] cryptd: max_cpu_qlen set to 1000 [ 0.195290] ACPI: Added _OSI(Module Device) [ 0.197014] ACPI: Added _OSI(Processor Device) [ 0.198009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204814] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209414] ACPI: Interpreter enabled [ 0.211057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213013] ACPI: Using IOAPIC for interrupt routing [ 0.214115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.217397] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228094] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.230034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242190] acpiphp: Slot [2] registered [ 0.243089] acpiphp: Slot [5] registered [ 0.244127] acpiphp: Slot [6] registered [ 0.245117] acpiphp: Slot [3] registered [ 0.247117] acpiphp: Slot [4] registered [ 0.248073] acpiphp: Slot [7] registered [ 0.249113] acpiphp: Slot [8] registered [ 0.251110] acpiphp: Slot [9] registered [ 0.252088] acpiphp: Slot [10] registered [ 0.254102] acpiphp: Slot [11] registered [ 0.256104] acpiphp: Slot [12] registered [ 0.257108] acpiphp: Slot [13] registered [ 0.259109] acpiphp: Slot [14] registered [ 0.260101] acpiphp: Slot [15] registered [ 0.262094] acpiphp: Slot [16] registered [ 0.263116] acpiphp: Slot [17] registered [ 0.265067] acpiphp: Slot [18] registered [ 0.266105] acpiphp: Slot [19] registered [ 0.268103] acpiphp: Slot [20] registered [ 0.269103] acpiphp: Slot [21] registered [ 0.271102] acpiphp: Slot [22] registered [ 0.272125] acpiphp: Slot [23] registered [ 0.274155] acpiphp: Slot [24] registered [ 0.275123] acpiphp: Slot [25] registered [ 0.277104] acpiphp: Slot [26] registered [ 0.279102] acpiphp: Slot [27] registered [ 0.280106] acpiphp: Slot [28] registered [ 0.282139] acpiphp: Slot [29] registered [ 0.284107] acpiphp: Slot [30] registered [ 0.285072] acpiphp: Slot [31] registered [ 0.286059] PCI host bridge to bus 0000:00 [ 0.288021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.292019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.294027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303930] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.307424] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.318112] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.323016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328583] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331086] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.334045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336778] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.350012] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.354014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.359364] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362693] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.366013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.376014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.384624] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.390014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.395014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.416019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.429259] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.431379] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.434440] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.437440] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.440200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.445120] iommu: Default domain type: Passthrough [ 0.447413] SCSI subsystem initialized [ 0.449161] ACPI: bus type USB registered [ 0.450097] usbcore: registered new interface driver usbfs [ 0.452094] usbcore: registered new interface driver hub [ 0.454087] usbcore: registered new device driver usb [ 0.455158] pps_core: LinuxPPS API ver. 1 registered [ 0.457011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.461067] PTP clock support registered [ 0.463132] EDAC MC: Ver: 3.0.0 [ 0.465065] PCI: Using ACPI for IRQ routing [ 0.466614] NetLabel: Initializing [ 0.467007] NetLabel: domain hash size = 128 [ 0.467941] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.469056] NetLabel: unlabeled traffic allowed by default [ 0.471102] vgaarb: loaded [ 0.472305] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.474007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.479291] clocksource: Switched to clocksource kvm-clock [ 0.583860] VFS: Disk quotas dquot_6.6.0 [ 0.585760] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.588544] *** VALIDATE ramfs *** [ 0.589823] *** VALIDATE hugetlbfs *** [ 0.591338] pnp: PnP ACPI init [ 0.593681] pnp: PnP ACPI: found 6 devices [ 0.612422] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.615646] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.617182] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.618693] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.620259] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.622702] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.625408] NET: Registered protocol family 2 [ 0.627748] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.632877] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.636407] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.641919] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.645199] TCP: Hash tables configured (established 65536 bind 65536) [ 0.648112] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.651488] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.654543] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657448] NET: Registered protocol family 1 [ 0.661225] RPC: Registered named UNIX socket transport module. [ 0.663757] RPC: Registered udp transport module. [ 0.665522] RPC: Registered tcp transport module. [ 0.667071] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.669290] NET: Registered protocol family 44 [ 0.670790] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.673219] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.674990] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.677059] PCI: CLS 0 bytes, default 64 [ 0.679116] Unpacking initramfs... [ 2.064752] debug: unmapping init [mem 0xffff9a45bcc64000-0xffff9a45bffcffff] [ 2.068923] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.071588] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.074756] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.564843] Initialise system trusted keyrings [ 2.566750] Key type blacklist registered [ 2.568837] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.581579] zbud: loaded [ 2.586383] *** VALIDATE nfs *** [ 2.587749] *** VALIDATE nfs4 *** [ 2.607418] pstore: using deflate compression [ 2.610598] Platform Keyring initialized [ 2.723630] NET: Registered protocol family 38 [ 2.725948] Key type asymmetric registered [ 2.728064] Asymmetric key parser 'x509' registered [ 2.730258] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.734063] io scheduler mq-deadline registered [ 2.736083] io scheduler kyber registered [ 2.737989] io scheduler bfq registered [ 2.740177] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.743826] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.747103] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.749875] ACPI: Power Button [PWRF] [ 2.756110] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.762907] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.772044] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.801505] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.830189] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.837245] Non-volatile memory driver v1.3 [ 2.839195] Linux agpgart interface v0.103 [ 2.871679] virtio_blk virtio1: [vda] 134864 512-byte logical blocks (69.1 MB/65.9 MiB) [ 2.874630] vda: detected capacity change from 0 to 69050368 [ 2.891652] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.895110] vdb: detected capacity change from 0 to 1073741824 [ 2.903137] libphy: Fixed MDIO Bus: probed [ 2.911725] usbcore: registered new interface driver usbserial_generic [ 2.915896] usbserial: USB Serial support registered for generic [ 2.918057] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.922395] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.924279] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.926985] mousedev: PS/2 mouse device common for all mice [ 2.930651] rtc_cmos 00:05: RTC can wake from S4 [ 2.933924] rtc_cmos 00:05: registered as rtc0 [ 2.935962] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.939266] intel_pstate: CPU model not supported [ 2.941188] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.946606] hid: raw HID events driver (C) Jiri Kosina [ 2.948341] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.950289] usbcore: registered new interface driver usbhid [ 2.953841] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.955683] usbhid: USB HID core driver [ 2.960187] drop_monitor: Initializing network drop monitor service [ 2.962846] Initializing XFRM netlink socket [ 2.965458] NET: Registered protocol family 10 [ 2.969180] Segment Routing with IPv6 [ 2.970875] NET: Registered protocol family 17 [ 2.973365] mpls_gso: MPLS GSO support [ 2.979529] RAS: Correctable Errors collector initialized. [ 2.981645] AVX version of gcm_enc/dec engaged. [ 2.983426] AES CTR mode by8 optimization enabled [ 3.062895] sched_clock: Marking stable (3062865391, 0)->(3978276867, -915411476) [ 3.066907] registered taskstats version 1 [ 3.068991] Loading compiled-in X.509 certificates [ 3.071202] zswap: loaded using pool lzo/zbud [ 3.096933] Key type big_key registered [ 3.110191] Key type encrypted registered [ 3.112172] ima: No TPM chip found, activating TPM-bypass! [ 3.114347] ima: Allocated hash algorithm: sha1 [ 3.116364] ima: No architecture policies found [ 3.118154] evm: Initialising EVM extended attributes: [ 3.120138] evm: security.selinux [ 3.121379] evm: security.ima [ 3.122399] evm: security.capability [ 3.123913] evm: HMAC attrs: 0x1 [ 3.126783] rtc_cmos 00:05: setting system clock to 2026-06-01 17:40:45 UTC (1780335645) [ 3.135220] debug: unmapping init [mem 0xffffffffa4c03000-0xffffffffa4dfffff] [ 3.138837] debug: unmapping init [mem 0xffffffffa3982000-0xffffffffa3c58fff] [ 3.147147] Write protecting the kernel read-only data: 28672k [ 3.150877] debug: unmapping init [mem 0xffffffffa2003000-0xffffffffa21fffff] [ 3.154243] debug: unmapping init [mem 0xffffffffa2914000-0xffffffffa29fffff] [ 3.186831] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.196919] systemd[1]: Detected virtualization kvm. [ 3.199271] systemd[1]: Detected architecture x86-64. [ 3.201594] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.229473] systemd[1]: No hostname configured. [ 3.231596] systemd[1]: Set hostname to . [ 3.234211] random: systemd: uninitialized urandom read (16 bytes read) [ 3.237149] systemd[1]: Initializing machine ID from random generator. [ 3.286730] random: ln: uninitialized urandom read (6 bytes read) [ 3.374093] random: systemd: uninitialized urandom read (16 bytes read) [ 3.376225] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.379905] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.385481] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.063469] device-mapper: uevent: version 1.0.3 [ 4.066076] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [[ 4.389427] random: fast init done  OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.750211] virtio_net virtio0 ens2: renamed from eth0 [ 4.789193] scsi host0: ata_piix [ 4.805481] scsi host1: ata_piix [ 4.807304] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.809407] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.579618] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.628960] random: crng init done [ 9.630769] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.027133] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.158886] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.385276] SELinux: Disabled at runtime. [ 11.442498] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.452271] systemd[1]: Detected virtualization kvm. [ 11.454506] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.928131] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.930991] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.935896] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.940157] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.944159] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.953201] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.963133] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ 11.991035] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.445459] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.707930] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.710670] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.817528] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.838763] EDAC sbridge: Ver: 1.1.2 [ 13.779221] Key type dns_resolver registered [ 14.081323] NFS: Registering the id_resolver key type [ 14.083761] Key type id_resolver registered [ 14.085459] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-client login: [ 47.855638] libcfs: loading out-of-tree module taints kernel. [ 47.905853] Key type ._llcrypt registered [ 47.914597] Key type .llcrypt registered [ 48.577944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.594958] alg: No test for adler32 (adler32-zlib) [ 50.352686] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 51.033563] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 52.743384] Key type lgssc registered [ 54.585284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 181.248740] Lustre: Mounted lustre-client [ 185.975391] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 198.562966] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing check_logdir /tmp/testlogs/ [ 202.785791] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing yml_node [ 206.140067] Lustre: DEBUG MARKER: Client: 2.17.53.34 [ 206.815663] Lustre: lustre-OST0000-osc-ffff9a4607e9b800: disconnect after 24s idle [ 208.634881] Lustre: DEBUG MARKER: MDS: 2.17.53.34 [ 211.259734] Lustre: DEBUG MARKER: OSS: 2.17.53.34 [ 212.922442] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Jun 1 13:44:13 EDT 2026 [ 229.209854] LustreError: 7924:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9a4607e9b800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 229.250680] Lustre: Unmounted lustre-client [ 249.489357] Key type lgssc unregistered [ 249.744612] LNet: 8370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 249.750525] LNetError: 8370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 249.774681] LNet: Removed LNI 192.168.203.57@tcp [ 250.890284] Key type .llcrypt unregistered [ 250.898521] Key type ._llcrypt unregistered [ 276.888960] Key type ._llcrypt registered [ 276.891432] Key type .llcrypt registered [ 277.118707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 277.131894] alg: No test for adler32 (adler32-zlib) [ 277.998945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 278.086705] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 278.108426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 281.827382] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 282.076296] LNet: 9250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.086155] LNetError: 9250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.113468] LNet: Removed LNI 192.168.203.57@tcp [ 282.794368] Key type .llcrypt unregistered [ 282.799377] Key type ._llcrypt unregistered [ 288.124949] Key type ._llcrypt registered [ 288.128955] Key type .llcrypt registered [ 289.393733] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 13:45:30 (1780335930) [ 289.670729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 289.685409] alg: No test for adler32 (adler32-zlib) [ 290.611512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.678304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.699413] LNet: 9661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.721585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.762753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 290.780918] LNet: 9668:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 296.935994] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 13:45:38 (1780335938) [ 297.153819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.173052] LNet: 10245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.186580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.240888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 303.231766] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 13:45:44 (1780335944) [ 303.340383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.360375] LNet: 10829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 303.389492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.443423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 309.271533] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 13:45:50 (1780335950) [ 309.348732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.385560] LNet: 11407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.416258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.517743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 316.113884] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 13:45:57 (1780335957) [ 316.390535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.435474] LNet: 12000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.452493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.540593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 317.085506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 323.520565] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 13:46:04 (1780335964) [ 323.593775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.631219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.721440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 324.044645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 324.418199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 330.102590] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 13:46:11 (1780335971) [ 330.214832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.235746] LNet: 13181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.242173] LNet: 13181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 330.273034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.358686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 335.922677] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 13:46:17 (1780335977) [ 336.072206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.102300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.150628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 336.183193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 336.545862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 336.586899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 336.924143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 336.954248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 337.365450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 337.400973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 337.860243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 337.898969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 338.167601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 338.199250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 338.637819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 338.668775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 339.096718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 339.143827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 339.598636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 339.639195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 339.949982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 339.988938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 340.451736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 340.490942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 340.977726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 341.021797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 341.386904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 341.452616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 348.622565] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 13:46:29 (1780335989) [ 348.780284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.807868] LNet: 14446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.813035] LNet: 14446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 348.853886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.020866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 349.079341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 356.600260] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 13:46:37 (1780335997) [ 356.761582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.833672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.002884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 357.065169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 363.670816] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 13:46:44 (1780336004) [ 363.891246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.921125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.987089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 364.026169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 369.637885] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 13:46:50 (1780336010) [ 369.788377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.846541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.927912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 369.977358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 376.876443] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 13:46:57 (1780336017) [ 376.977070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.022642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.100631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 377.155870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 382.881449] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 13:47:04 (1780336024) [ 383.080257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.108109] LNet: 17385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.113396] LNet: 17385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 383.139826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.204249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 383.245582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 388.775880] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 13:47:10 (1780336030) [ 388.881839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.918744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.988493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 389.238443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 394.935562] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 13:47:15 (1780336035) [ 395.058510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.121181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.326185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 395.549843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 401.107815] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 13:47:22 (1780336042) [ 401.311865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.348380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.500910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 401.747869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 407.841925] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 13:47:29 (1780336049) [ 407.922542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.996124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.468912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.750448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 414.257749] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 13:47:35 (1780336055) [ 414.439909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.478177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.537546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.579296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.808304] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 13:47:41 (1780336061) [ 419.874134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.929398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.037635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.082185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 425.483368] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 13:47:46 (1780336066) [ 425.559527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.591779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.670517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 431.539883] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 13:47:52 (1780336072) [ 431.677318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.717910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.799919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 438.548256] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 13:47:59 (1780336079) [ 438.848358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.896451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.009996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 439.432600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 444.861236] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 13:48:05 (1780336085) [ 444.955477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.011569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.074687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 445.128577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 450.557483] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 13:48:11 (1780336091) [ 450.674689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.706782] LNet: 23995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.712609] LNet: 23995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 450.742983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.800787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 450.828336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 456.745808] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 13:48:17 (1780336097) [ 456.869438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.931817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.017324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 457.070067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 465.327500] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 13:48:25 (1780336105) [ 465.414391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.455593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.570468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 465.613989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 471.668774] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 13:48:32 (1780336112) [ 471.805723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.861754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.922713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 472.006577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 472.037738] LNetError: 25768:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 472.092032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 472.127306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 478.112454] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 13:48:39 (1780336119) [ 478.224588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.331181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.466289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rZgCbPZ40Q [ 478.594584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rZgCbPZ40Q [ 478.660488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.rZgCbPZ40Q [ 479.040953] hrtimer: interrupt took 4210507 ns [ 484.057575] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 13:48:45 (1780336125) [ 484.347980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.389686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.453329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 484.497585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 484.550736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 489.867644] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 13:48:51 (1780336131) [ 490.076677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.117494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.218599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 492.958630] LNet: Message forwarding will be enabled in 2s [ 496.031128] LNet: Message forwarding enabled [ 496.386661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 496.429565] LNet: Message forwarding will stop in 2s to 6s [ 500.949812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 501.752034] LNet: Message forwarding disabled [ 503.497501] LNet: Message forwarding will be enabled in 2s [ 506.911321] LNet: Message forwarding enabled [ 509.367436] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 13:49:10 (1780336150) [ 509.572993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.649988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.740378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 515.076837] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 13:49:16 (1780336156) [ 515.259171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.294778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.336943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 515.365915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 515.409481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 515.444273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 515.475645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 515.512371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 515.531597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 515.555591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 515.582800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 515.611182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 515.639602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 515.692638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 515.728476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 515.764316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 515.792194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 515.819620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 515.841815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 515.872551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 515.899797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 515.915286] LNetError: 28838:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 521.121727] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 13:49:22 (1780336162) [ 521.240837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 521.286182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 521.356944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.999985] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 13:49:28 (1780336168) [ 528.836478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg357-server [ 530.728835] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 13:49:31 (1780336171) [ 530.817347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.882866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.963350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 530.988059] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 532.225375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[58-62]@tcp [ 532.245485] LNet: 30553:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 532.278766] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 532.302623] LNetError: 30342:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.58@tcp has gone from up to down [ 533.295626] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 533.311903] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 534.321260] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 534.339321] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 535.330504] LNetError: 30554:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.61@tcp has gone from up to down [ 535.352409] LNetError: 30554:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 536.362582] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 536.378611] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 538.683906] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 13:49:39 (1780336179) [ 540.468633] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.58@tcp is being used as a gateway but routing feature is not turned on [ 540.491225] LNetError: 30342:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 540.787317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg357-server [ 542.210203] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 13:49:43 (1780336183) [ 542.512167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.540775] LNetError: 31488:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.558924] LNet: Removed LNI 192.168.203.57@tcp [ 542.591776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.657548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 542.695685] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 543.363828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[58-63/2]@tcp [ 543.392907] LNet: 31605:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 543.400150] LNet: 31605:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 543.419911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[58-63/2]@tcp [ 543.427693] LNetError: 31492:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.58@tcp has gone from up to down [ 543.458766] LNetError: 31492:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 548.751854] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 13:49:50 (1780336190) [ 549.023689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.054976] LNetError: 32198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 549.089750] LNet: Removed LNI 192.168.203.57@tcp [ 549.113673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.198443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 549.247991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.278301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.331390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.388751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.446429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.500477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 549.560691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.597687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.663829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.730803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.758941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.805467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 549.883776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.917154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.982297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 550.050951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.091051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.135597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 550.219510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.259289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.342389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 550.421643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.460679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.510662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 550.583747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.614715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.671259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 550.747169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.787844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.838183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 550.889588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.920965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.966858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 556.470445] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 13:49:57 (1780336197) [ 558.293660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg357-server [ 559.873197] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 13:50:00 (1780336200) [ 561.471637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg357-server [ 563.280750] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 13:50:04 (1780336204) [ 564.130818] Key type .llcrypt unregistered [ 564.133041] Key type ._llcrypt unregistered [ 568.820982] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 568.947672] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 568.958548] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 568.965566] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 571.412529] Key type ._llcrypt registered [ 571.416859] Key type .llcrypt registered [ 571.632667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 571.645752] alg: No test for adler32 (adler32-zlib) [ 572.568779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.633360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 572.676481] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 573.211792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 573.231102] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 574.929536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 574.957200] LNet: Removed LNI 10.1.2.3@tcp [ 575.062973] LNet: 34320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 575.070277] LNetError: 34320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 575.086687] LNet: Removed LNI 192.168.203.57@tcp [ 575.595495] Key type .llcrypt unregistered [ 575.600608] Key type ._llcrypt unregistered [ 584.064414] Key type ._llcrypt registered [ 584.067159] Key type .llcrypt registered [ 584.927495] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 13:50:26 (1780336226) [ 585.301396] Key type .llcrypt unregistered [ 585.304360] Key type ._llcrypt unregistered [ 589.642846] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 589.750748] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 589.753963] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 589.757106] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 592.209874] Key type ._llcrypt registered [ 592.211740] Key type .llcrypt registered [ 592.386951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 592.406584] alg: No test for adler32 (adler32-zlib) [ 593.303185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 593.340855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 593.357239] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 593.893503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 593.909236] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 596.090653] LNet: 35812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.103313] LNetError: 35812:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.106912] LNet: Removed LNI 192.168.203.57@tcp [ 596.121313] LNet: Removed LNI 10.1.2.3@tcp [ 596.741257] Key type .llcrypt unregistered [ 596.744880] Key type ._llcrypt unregistered [ 606.364354] Key type ._llcrypt registered [ 606.367633] Key type .llcrypt registered [ 607.325475] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 13:50:48 (1780336248) [ 607.768378] Key type .llcrypt unregistered [ 607.769949] Key type ._llcrypt unregistered [ 611.842633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 611.948116] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 611.953245] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 611.956724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 614.377181] Key type ._llcrypt registered [ 614.380620] Key type .llcrypt registered [ 614.526715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 614.542158] alg: No test for adler32 (adler32-zlib) [ 615.407605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.443416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 615.463935] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 617.013025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 617.041765] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 618.644505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 618.670488] LNet: Removed LNI 10.1.2.3@tcp [ 618.693308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 618.730095] LNet: Removed LNI 10.1.2.31@tcp [ 618.750277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 618.775473] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.483348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 619.499970] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 620.448906] LNet: 37730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.455891] LNetError: 37730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 620.470688] LNet: Removed LNI 10.1.2.3@tcp [ 620.486344] LNet: Removed LNI 10.1.2.31@tcp [ 621.022584] Key type .llcrypt unregistered [ 621.026433] Key type ._llcrypt unregistered [ 630.509734] Key type ._llcrypt registered [ 630.511960] Key type .llcrypt registered [ 631.535502] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 13:51:12 (1780336272) [ 631.928649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 631.955974] alg: No test for adler32 (adler32-zlib) [ 632.856884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.896509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.928992] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 633.619813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 633.637857] LNet: 38682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.647514] LNetError: 38682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.659268] LNet: Removed LNI 192.168.203.57@tcp [ 633.687209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.726517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.57@tcp [ 633.749275] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 634.355632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.57@tcp [ 634.382565] LNet: Removed LNI 192.168.203.57@tcp [ 634.442527] LNet: 38817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.992416] Key type .llcrypt unregistered [ 635.003198] Key type ._llcrypt unregistered [ 645.554826] Key type ._llcrypt registered [ 645.557262] Key type .llcrypt registered [ 646.828861] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 13:51:27 (1780336287) [ 648.387802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg357-server [ 649.847520] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 13:51:31 (1780336291) [ 650.446488] Key type .llcrypt unregistered [ 650.450154] Key type ._llcrypt unregistered [ 656.138026] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 656.246315] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 656.257938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 656.271486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 658.846684] Key type ._llcrypt registered [ 658.851072] Key type .llcrypt registered [ 659.006706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 659.020949] alg: No test for adler32 (adler32-zlib) [ 659.922445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.975794] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 660.632683] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 662.612686] LNet: 40655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.625730] LNetError: 40655:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.633648] LNet: Removed LNI 192.168.203.57@tcp1 [ 662.646771] LNet: Removed LNI 10.1.2.3@tcp2 [ 663.193442] Key type .llcrypt unregistered [ 663.196985] Key type ._llcrypt unregistered [ 672.666532] Key type ._llcrypt registered [ 672.669499] Key type .llcrypt registered [ 673.937508] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 13:51:54 (1780336314) [ 674.237994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.249169] alg: No test for adler32 (adler32-zlib) [ 675.118503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.195407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 675.233444] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 676.165294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.189149] LNet: 41623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.195788] LNetError: 41623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.211182] LNet: Removed LNI 192.168.203.57@tcp [ 676.226407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.295871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 676.326162] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 682.574777] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 13:52:03 (1780336323) [ 682.665769] LNet: 42207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.677404] LNetError: 42207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.695173] LNet: Removed LNI 192.168.203.57@tcp [ 683.251784] Key type .llcrypt unregistered [ 683.254685] Key type ._llcrypt unregistered [ 687.777476] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.854225] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.863397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.871066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 690.368793] Key type ._llcrypt registered [ 690.371315] Key type .llcrypt registered [ 690.556610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 690.570072] alg: No test for adler32 (adler32-zlib) [ 691.444929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.489056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 691.516799] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 692.060465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.072397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 693.948869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 693.969208] LNet: 42958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.974330] LNetError: 42958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.978758] LNet: Removed LNI 192.168.203.57@tcp [ 693.987976] LNet: Removed LNI 10.1.2.3@tcp [ 694.011315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.070618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 694.109078] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 694.113376] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.518480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.543782] LNet: 42985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.550709] LNetError: 42985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.556437] LNetError: 42985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 694.562626] LNet: Removed LNI 192.168.203.57@tcp [ 694.571796] LNet: Removed LNI 10.1.2.3@tcp [ 694.596090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.659366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 694.679124] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 694.690022] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.168780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 695.197248] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 696.102824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.123948] LNet: 43147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.136403] LNetError: 43147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.143033] LNetError: 43147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 696.148319] LNet: Removed LNI 192.168.203.57@tcp [ 696.150640] LNet: Removed LNI 10.1.2.3@tcp [ 696.165562] LNet: Removed LNI 192.168.203.57@tcp2 [ 696.197433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.250227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 696.270083] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 696.274138] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 696.279275] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 702.616435] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 13:52:23 (1780336343) [ 702.726332] LNet: 43737:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.735267] LNetError: 43737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.747693] LNetError: 43737:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 702.759918] LNet: Removed LNI 192.168.203.57@tcp [ 702.762779] LNet: Removed LNI 10.1.2.3@tcp [ 702.777364] LNet: Removed LNI 192.168.203.57@tcp2 [ 703.470207] Key type .llcrypt unregistered [ 703.473534] Key type ._llcrypt unregistered [ 708.405470] Key type ._llcrypt registered [ 708.408520] Key type .llcrypt registered [ 708.544859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 708.555376] alg: No test for adler32 (adler32-zlib) [ 709.469360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 709.517659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 709.537706] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 710.182488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.218417] LNet: 44140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.234354] LNetError: 44140:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.253398] LNet: Removed LNI 192.168.203.57@tcp [ 710.278409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.346292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 710.366462] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 710.858327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.873817] LNet: 44170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.879704] LNetError: 44170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.897284] LNet: Removed LNI 192.168.203.57@tcp [ 710.921387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.002543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 711.044197] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 711.428637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 711.456899] LNet: Removed LNI 192.168.203.57@tcp [ 711.471587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.532567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 711.557761] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 712.021204] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.112716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.117251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.120654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.534683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.580925] LNet: 44278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.586439] LNet: 44278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 714.595577] LNetError: 44278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.599906] LNetError: 44278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 714.624251] LNet: Removed LNI 192.168.203.57@tcp [ 714.642954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.709352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 714.729971] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 720.458637] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:52:41 (1780336361) [ 720.546059] LNet: 44866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.550587] LNetError: 44866:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.566480] LNet: Removed LNI 192.168.203.57@tcp [ 721.105841] Key type .llcrypt unregistered [ 721.108417] Key type ._llcrypt unregistered [ 726.631776] Key type ._llcrypt registered [ 726.633912] Key type .llcrypt registered [ 726.835250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.855225] alg: No test for adler32 (adler32-zlib) [ 727.825631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.872187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 727.894088] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 728.578165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.597767] LNet: 45274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.606663] LNetError: 45274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.624844] LNet: Removed LNI 192.168.203.57@tcp [ 728.665411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.736000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 728.761188] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 734.644368] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:52:55 (1780336375) [ 734.883288] LNet: 45867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.893083] LNetError: 45867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.914422] LNet: Removed LNI 192.168.203.57@tcp [ 735.577816] Key type .llcrypt unregistered [ 735.581914] Key type ._llcrypt unregistered [ 740.338141] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 740.482068] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 740.489311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 740.491747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 742.988789] Key type ._llcrypt registered [ 742.991201] Key type .llcrypt registered [ 743.151347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.167723] alg: No test for adler32 (adler32-zlib) [ 744.037161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.096277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.124118] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 744.801348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.816192] LNet: 46323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.820915] LNetError: 46323:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.834775] LNet: Removed LNI 192.168.203.57@tcp [ 744.864385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.922613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 744.946765] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 751.006880] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:53:11 (1780336391) [ 751.106031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.142352] LNet: 46914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 751.149221] LNetError: 46914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 751.166129] LNet: Removed LNI 192.168.203.57@tcp [ 751.216358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.368684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 757.970104] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:53:18 (1780336398) [ 758.164204] LNet: 47502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.801677] Key type .llcrypt unregistered [ 758.810239] Key type ._llcrypt unregistered [ 766.575306] Key type ._llcrypt registered [ 766.577919] Key type .llcrypt registered [ 766.845461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 766.859792] alg: No test for adler32 (adler32-zlib) [ 767.882814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.942204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 767.983628] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 769.142116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.180652] LNet: 47901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.205597] LNetError: 47901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.254397] LNet: Removed LNI 192.168.203.57@tcp [ 769.288271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.370938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 769.420636] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 776.692195] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:53:37 (1780336417) [ 776.826101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.862934] LNet: 48490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.874758] LNetError: 48490:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.913475] LNet: Removed LNI 192.168.203.57@tcp [ 776.962604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.067594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 783.448968] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:53:44 (1780336424) [ 785.134922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 786.900278] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:53:47 (1780336427) [ 789.131274] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 791.085570] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:53:51 (1780336431) [ 792.534222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 794.798247] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:53:55 (1780336435) [ 797.020357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 798.933915] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:53:59 (1780336439) [ 800.615510] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 803.258761] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 13:54:03 (1780336443) [ 803.437681] LNet: 50537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.537896] Key type .llcrypt unregistered [ 804.542559] Key type ._llcrypt unregistered [ 811.932939] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.172103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.187098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.200016] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.706819] Key type ._llcrypt registered [ 814.709725] Key type .llcrypt registered [ 814.924683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.954125] alg: No test for adler32 (adler32-zlib) [ 815.898992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.954083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 815.975094] LNet: Added LNI 192.168.203.57@tcp [8/512/0/180] [ 815.980758] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 815.983683] LNet: 50867:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 816.070541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.110078] LNet: 50886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.126975] LNetError: 50886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.136684] LNet: Removed LNI 192.168.203.57@tcp [ 816.146324] LNetError: 50886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 816.166602] LNet: Removed LNI 10.1.2.3@tcp [ 816.199875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.373976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 816.415325] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 816.545734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.577530] LNet: 50912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.585285] LNetError: 50912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.617406] LNet: Removed LNI 192.168.203.57@tcp [ 816.638968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.724201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 816.741480] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 816.763409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 816.769195] LNet: 50920:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 816.881520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.918842] LNet: 50938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.932112] LNetError: 50938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.940969] LNet: Removed LNI 192.168.203.57@tcp [ 816.942920] LNetError: 50938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 816.964088] LNet: Removed LNI 10.1.2.3@tcp [ 817.011792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.104456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 817.131371] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 817.136709] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 817.139715] LNet: 50946:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 817.393653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.452766] LNet: 50972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.460023] LNetError: 50972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.482519] LNet: Removed LNI 192.168.203.57@tcp [ 817.490587] LNetError: 50972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 817.520369] LNet: Removed LNI 10.1.2.3@tcp [ 817.577126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.713299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 817.742916] LNet: Added LNI 192.168.203.57@tcp [8/256/0/182] [ 817.749463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 817.919889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.961899] LNet: 51000:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.977214] LNetError: 51000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.990424] LNet: Removed LNI 192.168.203.57@tcp [ 817.992681] LNetError: 51000:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.029058] LNet: Removed LNI 10.1.2.3@tcp [ 818.065902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.127931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 818.170541] LNet: Added LNI 192.168.203.57@tcp [8/256/0/182] [ 818.175181] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 818.178708] LNet: 51008:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 818.291408] LNet: Removed LNI 192.168.203.57@tcp [ 818.308553] LNet: Removed LNI 10.1.2.3@tcp [ 819.181849] Key type .llcrypt unregistered [ 819.186478] Key type ._llcrypt unregistered [ 830.360678] Key type ._llcrypt registered [ 830.363031] Key type .llcrypt registered [ 831.512584] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:54:32 (1780336472) [ 831.939721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 831.959029] alg: No test for adler32 (adler32-zlib) [ 832.882392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.946389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 832.990111] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 833.735241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.57 [ 833.751825] LNet: 51984:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 833.759838] LNet: Removed LNI 192.168.203.57@tcp [ 839.611736] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:54:40 (1780336480) [ 839.782571] LNet: 52549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.805753] LNetError: 52549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.852888] LNet: Removed LNI 192.168.203.57@tcp [ 840.650387] Key type .llcrypt unregistered [ 840.653244] Key type ._llcrypt unregistered [ 847.300038] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 847.537440] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 850.492442] Key type ._llcrypt registered [ 850.495775] Key type .llcrypt registered [ 850.681240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 850.721524] alg: No test for adler32 (adler32-zlib) [ 852.402818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 858.867778] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:54:59 (1780336499) [ 858.956094] LNet: 53460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.971991] LNetError: 53460:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.001218] LNet: Removed LNI 10.1.2.3@tcp [ 859.621892] Key type .llcrypt unregistered [ 859.631231] Key type ._llcrypt unregistered [ 866.335734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 866.593724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 866.608868] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 866.612934] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 869.288587] Key type ._llcrypt registered [ 869.300710] Key type .llcrypt registered [ 869.462654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 869.474163] alg: No test for adler32 (adler32-zlib) [ 870.456719] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.849740] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:55:17 (1780336517) [ 878.591661] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 880.423522] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:55:21 (1780336521) [ 880.768509] LNet: 54654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.780200] LNetError: 54654:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 880.823920] LNet: Removed LNI 10.1.2.3@tcp [ 881.560502] Key type .llcrypt unregistered [ 881.566101] Key type ._llcrypt unregistered [ 887.638176] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 887.938092] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 887.947865] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 887.956535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 890.737478] Key type ._llcrypt registered [ 890.742987] Key type .llcrypt registered [ 891.082856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.102301] alg: No test for adler32 (adler32-zlib) [ 892.169682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.270385] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 902.605359] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:55:43 (1780336543) [ 903.359796] LNet: 55887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.367044] LNetError: 55887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 903.403156] LNet: Removed LNI 10.1.2.3@tcp [ 904.818637] Key type .llcrypt unregistered [ 904.828117] Key type ._llcrypt unregistered [ 913.092389] Key type ._llcrypt registered [ 913.095495] Key type .llcrypt registered [ 913.605272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 913.624103] alg: No test for adler32 (adler32-zlib) [ 914.769634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 914.839114] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 924.588277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 925.509687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 925.569824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 925.639978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 925.702042] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 926.043973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.095266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 926.111944] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 926.140987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.185479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.220502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 926.285336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 926.327995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 926.366383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 926.419463] LNet: 56326:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.456356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 926.507572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 926.550645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 926.589242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 926.639665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.697819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.746386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 927.454790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 927.524433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 927.596666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 927.614481] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 927.619634] LNet: Skipped 1 previous similar message [ 927.913900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 927.960378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 927.995548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 928.050565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 928.140093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 928.199275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 928.243946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 928.309523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 928.327628] LNet: 56435:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.341079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 928.381669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 928.466378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 928.519523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 928.569689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 928.651718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 928.702686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 929.249292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 929.291722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 929.356450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 929.372620] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 929.648947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.683629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.715094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.755392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.803485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.830165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.856515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.878528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.892732] LNet: 56544:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 929.911433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.941747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.989714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.041301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.082828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.142517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.220931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 930.749456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 930.789442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 930.834376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 931.203334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 931.269409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 931.309955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 931.367807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 931.429339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 931.474614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 931.527347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 931.578365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 931.617597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 931.642592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 931.670538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 931.703963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 931.741832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 931.798813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 931.844810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 932.541138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 932.568741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 932.607288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 932.622814] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 932.630507] LNet: Skipped 1 previous similar message [ 932.887911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 932.940724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 932.973802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 933.031756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 933.109514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 933.171923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 933.226586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 933.285135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 933.324025] LNet: 56763:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 933.337646] LNet: 56763:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 933.363583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 933.402980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 933.444147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 933.490812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 933.532782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 933.577342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 933.642706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 934.447703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 934.493507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 934.559160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 934.810819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.871160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 934.918189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.947667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.993829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 935.036305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 935.078716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.141436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.218424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.278817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.304339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.333985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.358143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.418469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.490529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 935.828327] LNet: 56913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.847616] LNetError: 56913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.866741] LNet: Removed LNI 192.168.203.57@tcp [ 936.412816] Key type .llcrypt unregistered [ 936.416167] Key type ._llcrypt unregistered [ 952.771678] Key type ._llcrypt registered [ 952.773902] Key type .llcrypt registered [ 954.142687] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:56:34 (1780336594) [ 955.009494] Key type .llcrypt unregistered [ 955.013114] Key type ._llcrypt unregistered [ 961.008824] Key type ._llcrypt registered [ 961.014532] Key type .llcrypt registered [ 961.335607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.371403] alg: No test for adler32 (adler32-zlib) [ 962.382809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 962.441454] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 970.284758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 980.251350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.349079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 980.389795] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 982.197415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 982.248584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 1 [ 982.285567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 982.345827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 1 [ 982.416888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 982.478845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 1 [ 982.552735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 982.620378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 1 [ 982.709658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 982.770316] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 982.793406] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 983.098878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.128786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 983.171449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.205748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.259214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.302673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 983.351941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 983.401530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 983.426953] LNet: 58381:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 983.456755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 983.499753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 983.531567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 983.567565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 983.604369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.638780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 983.679216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 984.378518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 984.409927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 1 [ 984.438820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 984.467269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 1 [ 984.496119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 984.526450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 1 [ 984.556735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 984.580488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 1 [ 984.621804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 984.642736] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 984.653770] LNet: Skipped 2 previous similar messages [ 984.912398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 984.955889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 985.004429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 985.036448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.073877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.137478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.174764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.225820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.245562] LNet: 58503:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.287832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.336580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.385275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.434507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.488521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.540523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.598711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 986.117754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 986.149977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 1 [ 986.183376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 986.212560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 1 [ 986.241874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 986.293751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 1 [ 986.336530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 986.385858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 1 [ 986.451575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 986.478500] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 986.484456] LNet: Skipped 2 previous similar messages [ 986.705923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.748163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 986.789366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.819408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.863888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.907134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.950983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.986289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.011742] LNet: 58624:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 987.038691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.086416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.115133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.144146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.180093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.223629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.277658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 987.924159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 987.965952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 1 [ 988.008866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 988.057541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 1 [ 988.101932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 988.139667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 1 [ 988.172517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 988.198852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 1 [ 988.236420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 988.521886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.570770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 988.606119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.631880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.658520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.701041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.741773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.793750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.840823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.871453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.906494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.940541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.990573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.053046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.137717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 989.734508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 989.773680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 1 [ 989.810862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 989.860785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 1 [ 989.906793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 989.970078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 1 [ 990.032549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 990.109766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 1 [ 990.179686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 990.223129] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 990.235194] LNet: Skipped 5 previous similar messages [ 990.522641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.558757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 990.605658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.654683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.697931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.733890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.762773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.799574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.814237] LNet: 58867:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.822523] LNet: 58867:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 990.846766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.873342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.900625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.940219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.979631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 991.038695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 991.079715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 991.662318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 991.696472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_error -r 1 [ 991.734802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 991.775351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_error -r 1 [ 991.820278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 991.856833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_error -r 1 [ 991.921570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 991.971220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_error -r 1 [ 992.013254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 992.354921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 992.412251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 992.446363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 992.502617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 992.556624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 992.619132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 992.677663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.730921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.775804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.831519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.869808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.923788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.986736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.030870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 993.072136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 998.414924] LNet: 59064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.420926] LNetError: 59064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.429625] LNet: Removed LNI 192.168.203.57@tcp [ 998.442825] LNet: Removed LNI 192.168.203.57@tcp1 [ 998.823831] Key type .llcrypt unregistered [ 998.825666] Key type ._llcrypt unregistered [ 1016.810027] Key type ._llcrypt registered [ 1016.813140] Key type .llcrypt registered [ 1018.080561] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:57:39 (1780336659) [ 1019.028379] Key type .llcrypt unregistered [ 1019.032367] Key type ._llcrypt unregistered [ 1024.438274] Key type ._llcrypt registered [ 1024.441326] Key type .llcrypt registered [ 1024.631995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.648452] alg: No test for adler32 (adler32-zlib) [ 1025.548521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1025.579992] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1032.918842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1033.665576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 1033.701338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 1033.759993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1033.789371] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1034.027262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.054804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.077117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.102718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.133127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.169610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.195122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.239195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.260760] LNet: 60317:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.277728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.310908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.350168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.382839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.442525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.481323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.518137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.997446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 1035.033792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 1035.080115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1035.097886] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1035.334602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.379912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.418260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.472559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.504666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.536739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.569944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.613381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.628258] LNet: 60426:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.642580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.678595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.711995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.777245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.806248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.832846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.858530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1036.333957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 1036.391307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 1036.444121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1036.473393] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1036.706589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.752447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1036.785337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.819426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.863221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.891510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.919106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.959932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.981704] LNet: 60536:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1037.008987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.035649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.085387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.119589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.161078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.195799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.230326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1037.476465] LNet: 60577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.483053] LNetError: 60577:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1037.500096] LNet: Removed LNI 192.168.203.57@tcp [ 1037.900823] Key type .llcrypt unregistered [ 1037.905067] Key type ._llcrypt unregistered [ 1049.492284] Key type ._llcrypt registered [ 1049.494702] Key type .llcrypt registered [ 1050.568359] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:58:11 (1780336691) [ 1051.219273] Key type .llcrypt unregistered [ 1051.221302] Key type ._llcrypt unregistered [ 1056.442562] Key type ._llcrypt registered [ 1056.445450] Key type .llcrypt registered [ 1056.659384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.688973] alg: No test for adler32 (adler32-zlib) [ 1057.669739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1057.698907] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1064.728382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1074.632581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.710387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1074.763022] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1076.427198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 1076.468747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 1 [ 1076.556960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 1076.657512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 1 [ 1076.713517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 1076.752896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 1 [ 1076.799941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 1076.848228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 1 [ 1076.889732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1076.907125] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1076.913786] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 1077.185242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1077.231882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1077.294656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1077.343299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1077.381746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1077.432857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1077.486975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.548233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.593732] LNet: 62045:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.626961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.668729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.694847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.716466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.740985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.768799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.794474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1077.828845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.418459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 1078.463481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_error -r 1 [ 1078.498520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 1078.549244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_error -r 1 [ 1078.623578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 1078.696365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_error -r 1 [ 1078.764295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 1078.809575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_error -r 1 [ 1078.855891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1078.892459] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1078.903669] LNet: Skipped 2 previous similar messages [ 1079.131231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1079.161680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1079.188327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1079.217185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1079.248916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1079.276739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1079.302942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1079.332611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1079.350983] LNet: 62168:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1079.370838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1079.412373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1079.456794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1079.520262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1079.565218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1079.614660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1079.670412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1079.724571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1080.327322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 1080.384516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_timeout -r 1 [ 1080.468739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 1080.527425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_timeout -r 1 [ 1080.577423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 1080.625794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_timeout -r 1 [ 1080.662631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 1080.698954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_timeout -r 1 [ 1080.759663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 1080.784879] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1080.790433] LNet: Skipped 1 previous similar message [ 1081.044935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1081.088252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1081.158981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1081.220769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1081.289448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1081.356415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1081.396509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1081.443919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1081.472424] LNet: 62291:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1081.513490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1081.544459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1081.607208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1081.658397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1081.713746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1081.776489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1081.841176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1081.908729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1087.483619] LNet: 62370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.488393] LNetError: 62370:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.495730] LNet: Removed LNI 192.168.203.57@tcp [ 1087.505691] LNet: Removed LNI 192.168.203.57@tcp1 [ 1087.895826] Key type .llcrypt unregistered [ 1087.898910] Key type ._llcrypt unregistered [ 1104.793718] Key type ._llcrypt registered [ 1104.805876] Key type .llcrypt registered [ 1106.144537] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:59:06 (1780336746) [ 1107.840416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1109.969048] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:59:10 (1780336750) [ 1111.058428] Key type .llcrypt unregistered [ 1111.061944] Key type ._llcrypt unregistered [ 1118.243949] Key type ._llcrypt registered [ 1118.248680] Key type .llcrypt registered [ 1118.443420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.472219] alg: No test for adler32 (adler32-zlib) [ 1119.359509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1119.385110] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1127.690747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1128.486980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 1128.526832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 1128.563475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1128.586420] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1128.610218] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1128.857587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.900648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1128.935894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.974163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1129.030382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1129.080524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.130578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.162346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.189833] LNet: 63916:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.222924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.276971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.352833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.396323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.443936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.486065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1129.773504] LNet: 63955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.788401] LNetError: 63955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1130.853323] LNet: Removed LNI 192.168.203.57@tcp [ 1131.387449] Key type .llcrypt unregistered [ 1131.389298] Key type ._llcrypt unregistered [ 1150.185792] Key type ._llcrypt registered [ 1150.189721] Key type .llcrypt registered [ 1150.660547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.687623] alg: No test for adler32 (adler32-zlib) [ 1151.705920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1151.737985] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1160.305639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1170.736893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.826549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1170.870694] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1172.596695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 1172.645750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e network_timeout -r 1 [ 1172.712404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 1172.759448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e network_timeout -r 1 [ 1172.797493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 1172.844651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e network_timeout -r 1 [ 1172.879414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 1172.922164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e network_timeout -r 1 [ 1172.961298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1172.980755] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1173.227577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.261818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1173.279387] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1173.291072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.296345] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 1173.327533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.355046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.388829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.425901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.498522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.545183] LNet: 64836:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.587361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.644976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.703522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.739093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.774762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.819813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.324047] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1178.331773] LNet: Skipped 2 previous similar messages [ 1178.632477] LNet: 64910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.660321] LNetError: 64910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.671214] LNet: Removed LNI 192.168.203.57@tcp [ 1178.680838] LNet: Removed LNI 192.168.203.57@tcp1 [ 1179.200467] Key type .llcrypt unregistered [ 1179.203704] Key type ._llcrypt unregistered [ 1196.037102] Key type ._llcrypt registered [ 1196.043133] Key type .llcrypt registered [ 1197.517759] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:00:38 (1780336838) [ 1198.037937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1198.057791] alg: No test for adler32 (adler32-zlib) [ 1198.989673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1199.109611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1199.168804] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1200.125466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1200.184579] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1201.230552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1201.304696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1201.373232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1201.415181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1201.475442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1201.550960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1201.618206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 1201.654940] LNetError: 66010:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1202.143354] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1202.146673] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1204.191445] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1205.233762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1208.291175] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1208.778792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1208.833170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1208.930598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1208.952296] LNet: 66117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.961665] LNetError: 66117:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.971988] LNet: Removed LNI 192.168.203.57@tcp [ 1208.991877] LNet: Removed LNI 192.168.203.57@tcp1 [ 1209.021984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.126302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1209.152172] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1209.905106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1209.935121] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1210.963852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1211.026762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.100902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1211.139887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1211.187741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1211.235460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1211.300660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1211.355254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 1211.386296] LNetError: 66390:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1212.127352] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1215.034359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1218.272157] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1218.277059] LNet: Skipped 1 previous similar message [ 1223.026222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1223.075603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1223.116702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.122287] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:01:10 (1780336870) [ 1230.268909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1230.307839] LNet: 67113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.318529] LNetError: 67113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.328185] LNetError: 67113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1230.336740] LNet: Removed LNI 192.168.203.57@tcp [ 1230.363538] LNet: Removed LNI 192.168.203.57@tcp1 [ 1230.406270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.479678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1230.510799] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1231.364608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1231.392485] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1232.645832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1232.723348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1232.763663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1232.802622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1232.834881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1232.879609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1232.961777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 0 [ 1233.075819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1233.507301] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.57@tcp [ 1234.591662] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1234.605843] LNet: Skipped 1 previous similar message [ 1249.771457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1249.954752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1250.012139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.049543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 500 [ 1250.225563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1257.125753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1257.218218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1257.257386] LNet: 67619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1257.266391] LNetError: 67619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1257.276807] LNetError: 67619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1257.282721] LNet: Removed LNI 192.168.203.57@tcp [ 1257.296556] LNet: Removed LNI 192.168.203.57@tcp1 [ 1257.312407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.387251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1257.430336] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1258.439554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.503843] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1259.818424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1259.898640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.985695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1260.014475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1260.083868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1260.137451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1260.162621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1260.191732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 0 [ 1261.471519] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1261.485302] LNet: Skipped 3 previous similar messages [ 1262.608621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1271.974368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1272.011675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1272.056681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.049335] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:02:00 (1780336920) [ 1280.559286] LNet: 68665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.575035] LNetError: 68665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1280.590715] LNetError: 68665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1280.612901] LNet: Removed LNI 192.168.203.57@tcp [ 1280.638991] LNet: Removed LNI 192.168.203.57@tcp1 [ 1281.498724] Key type .llcrypt unregistered [ 1281.502108] Key type ._llcrypt unregistered [ 1287.929304] Key type ._llcrypt registered [ 1287.931555] Key type .llcrypt registered [ 1288.133417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.153858] alg: No test for adler32 (adler32-zlib) [ 1289.122858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1289.168592] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1298.537267] Lustre: DEBUG MARKER: Initial discovery [ 1298.615586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 1301.050734] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1301.186488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1301.239740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 1301.278398] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1301.327716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1301.388593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.157@tcp -r 1 -m GET -l 3 [ 1301.432294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.157@tcp -r 1 -m GET -e local_error [ 1301.471241] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1301.492256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.157@tcp -r 1 -m PUT -l 6 [ 1302.800292] Lustre: DEBUG MARKER: Force oleg357-client.virtnet to discover 192.168.203.157@tcp (in background) [ 1303.007196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 1304.509438] Lustre: DEBUG MARKER: Force oleg357-server to discover 192.168.203.57@tcp [ 1306.086663] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1307.433516] Lustre: DEBUG MARKER: Wait for 69428 [ 1309.300749] Lustre: DEBUG MARKER: Finished wait on 69428 [ 1309.402911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1309.497413] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1309.510956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1309.933230] LNet: 69876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.947384] LNetError: 69876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.971876] LNet: Removed LNI 192.168.203.57@tcp [ 1310.722574] Key type .llcrypt unregistered [ 1310.732166] Key type ._llcrypt unregistered [ 1328.515773] Key type ._llcrypt registered [ 1328.520895] Key type .llcrypt registered [ 1329.989559] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:02:50 (1780336970) [ 1330.445454] Key type .llcrypt unregistered [ 1330.449459] Key type ._llcrypt unregistered [ 1336.254174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1336.442244] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1336.448244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1336.458383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1339.035173] Key type ._llcrypt registered [ 1339.039386] Key type .llcrypt registered [ 1339.293162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.309391] alg: No test for adler32 (adler32-zlib) [ 1340.284231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.366165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.383725] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1341.348194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1341.363369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1350.437127] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:03:11 (1780336991) [ 1350.549660] LNet: 71944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.564118] LNetError: 71944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.579371] LNet: Removed LNI 192.168.203.57@tcp [ 1350.589665] LNet: Removed LNI 10.1.2.3@tcp [ 1351.657243] Key type .llcrypt unregistered [ 1351.663437] Key type ._llcrypt unregistered [ 1358.857300] Key type ._llcrypt registered [ 1358.861501] Key type .llcrypt registered [ 1359.095205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1359.113857] alg: No test for adler32 (adler32-zlib) [ 1360.074715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.169350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1360.229075] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1361.089965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1361.146880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1362.266277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.57@tcp [ 1363.360172] LNetError: Unexpected error -22 connecting to 192.168.203.57@tcp at host 192.168.203.57:988 [ 1369.565900] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:03:30 (1780337010) [ 1369.767912] LNet: 73080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.789456] LNetError: 73080:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.810262] LNet: Removed LNI 192.168.203.57@tcp [ 1369.841310] LNet: Removed LNI 10.1.2.3@tcp [ 1370.607567] Key type .llcrypt unregistered [ 1370.612416] Key type ._llcrypt unregistered [ 1376.711961] Key type ._llcrypt registered [ 1376.714097] Key type .llcrypt registered [ 1376.872355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1376.897356] alg: No test for adler32 (adler32-zlib) [ 1377.853895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.953385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1377.993127] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1378.873357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1378.900167] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1380.077693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.57@tcp1 --nid 192.168.203.57@tcp2 [ 1384.541575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp1 192.168.203.57@tcp2 [ 1384.558111] LNetError: 73920:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.57@tcp1 to 192.168.203.57@tcp2 - Route table may be misconfigured [ 1384.580916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp2 192.168.203.57@tcp1 [ 1391.153885] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:03:52 (1780337032) [ 1391.319910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1391.338383] LNet: 74487:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.346577] LNetError: 74487:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.359453] LNet: Removed LNI 192.168.203.57@tcp1 [ 1392.421508] LNet: Removed LNI 192.168.203.57@tcp2 [ 1392.505759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.591149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.638091] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1393.358154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1393.389909] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1394.252318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1394.315918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.57@tcp [ 1394.351288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp1 192.168.203.57@tcp1 [ 1394.405775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.57@tcp -d 192.168.203.57@tcp -e network_timeout [ 1394.461405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -e network_timeout [ 1394.509826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -e network_timeout [ 1394.556919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -e network_timeout [ 1394.594098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1394.617575] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1394.649176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1394.657081] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1394.755948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1394.858739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1401.404912] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:04:01 (1780337041) [ 1401.569789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.614258] LNet: 75354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.639540] LNetError: 75354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.653528] LNetError: 75354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1401.665411] LNet: Removed LNI 192.168.203.57@tcp [ 1402.736439] LNet: Removed LNI 192.168.203.57@tcp1 [ 1402.801817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.936168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1403.261173] LNet: 75393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.821577] Key type .llcrypt unregistered [ 1403.823539] Key type ._llcrypt unregistered [ 1416.145771] Key type ._llcrypt registered [ 1416.151243] Key type .llcrypt registered [ 1417.624250] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:04:18 (1780337058) [ 1418.029352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.044780] alg: No test for adler32 (adler32-zlib) [ 1419.073641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.735628] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1422.460550] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:04:23 (1780337063) [ 1422.570989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.607638] LNet: 76486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.665294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.758063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.790364] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1423.543552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1423.562577] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1424.423187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1424.462210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp1 [ 1424.495738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp1 [ 1430.816876] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:04:31 (1780337071) [ 1433.068260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg357-server [ 1434.967789] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:04:35 (1780337075) [ 1437.552756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg357-server [ 1439.628932] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:04:40 (1780337080) [ 1441.682839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg357-server [ 1444.360545] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:04:44 (1780337084) [ 1446.952969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg357-server [ 1449.050334] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:04:49 (1780337089) [ 1451.311706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg357-server [ 1453.235915] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:04:54 (1780337094) [ 1455.310907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg357-server [ 1457.426918] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:04:58 (1780337098) [ 1459.363636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg357-server [ 1461.023819] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:05:02 (1780337102) [ 1462.874156] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg357-server [ 1465.041530] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:05:05 (1780337105) [ 1467.036308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg357-server [ 1468.623739] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:05:09 (1780337109) [ 1468.707097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.755878] LNet: 80365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.767799] LNetError: 80365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.781915] LNet: Removed LNI 192.168.203.57@tcp [ 1469.799439] LNet: Removed LNI 192.168.203.57@tcp1 [ 1469.825939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.904386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.924528] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1470.723594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1470.787286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1470.902621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1470.921571] LNet: 80518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1470.927064] LNetError: 80518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.931736] LNetError: 80518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1471.974436] LNet: Removed LNI 192.168.203.57@tcp [ 1472.005768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.080308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1472.115591] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1472.822099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1473.003455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.031543] LNet: 80663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1473.045426] LNetError: 80663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1473.060195] LNet: Removed LNI 192.168.203.57@tcp [ 1473.090206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.142595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.160018] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1473.904729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1473.999875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1474.254690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.309557] LNet: Removed LNI 192.168.203.57@tcp [ 1475.363672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.416828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1475.446600] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1476.248746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1476.522629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1476.542351] LNetError: 80957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.547527] LNetError: 80957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1477.612172] LNet: Removed LNI 192.168.203.57@tcp [ 1477.702744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.771577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1477.802739] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1478.574451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1478.661635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1478.970843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.018645] LNet: 81104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.024978] LNet: 81104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1480.105847] LNet: Removed LNI 192.168.203.57@tcp [ 1480.131916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.233781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1480.265943] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1481.561225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1481.873561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.895767] LNetError: 81250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.904769] LNetError: 81250:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1481.920570] LNet: Removed LNI 192.168.203.57@tcp [ 1481.944538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.019933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1482.069619] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1483.161768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1483.261808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1484.874984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.927961] LNet: Removed LNI 192.168.203.57@tcp [ 1486.002067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.130270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1486.158324] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1487.224555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1487.522241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.570684] LNet: 81552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.577952] LNet: 81552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1488.622521] LNet: Removed LNI 192.168.203.57@tcp [ 1488.679903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.755546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1488.795385] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1489.836698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1489.924582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1490.188287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.217557] LNetError: 81699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.227916] LNetError: 81699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1490.246187] LNet: Removed LNI 192.168.203.57@tcp [ 1490.299120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.403326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1490.443052] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1491.582237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1494.429637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.530341] LNet: Removed LNI 192.168.203.57@tcp [ 1495.585852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.679390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.751667] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1496.879432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1496.975643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1497.326166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.410361] LNet: Removed LNI 192.168.203.57@tcp [ 1498.478881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1498.570806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1498.594112] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1499.552532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1500.011327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.095774] LNet: Removed LNI 192.168.203.57@tcp [ 1501.136373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.254052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.313078] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1502.113293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1502.172969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1502.198554] LNet: Removed LNI 192.168.203.57@tcp [ 1502.214981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.293693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.322578] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1503.131231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1510.157398] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:05:51 (1780337151) [ 1510.493827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.527941] LNet: 82993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.538176] LNet: 82993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1510.545822] LNetError: 82993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.553264] LNetError: 82993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1510.568353] LNet: Removed LNI 192.168.203.57@tcp [ 1510.608459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.686140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1510.716075] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1512.140523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.195734] LNet: Removed LNI 192.168.203.57@tcp231 [ 1512.224674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.293884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1512.317206] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1512.323098] LNet: 83139:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1512.767888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1512.814107] LNet: Removed LNI 192.168.203.57@tcp231 [ 1512.838460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1512.864288] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1514.010763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1514.063272] LNet: Removed LNI 192.168.203.57@tcp231 [ 1514.081945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1514.189199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1514.225664] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/0] [ 1514.229585] LNet: 83292:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1514.625056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1514.657933] LNet: Removed LNI 192.168.203.57@tcp231 [ 1514.675710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1514.709789] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/0] [ 1515.958619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.013864] LNet: Removed LNI 192.168.203.57@tcp231 [ 1516.069477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.172272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1516.197225] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/60] [ 1516.209918] LNet: 83443:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1516.730956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1516.774591] LNet: Removed LNI 192.168.203.57@tcp231 [ 1516.801717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1516.842681] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/60] [ 1518.472208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1518.518221] LNet: Removed LNI 192.168.203.57@tcp231 [ 1518.567097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.649154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1518.682383] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1518.692191] LNet: 83595:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1519.507512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1519.558371] LNet: Removed LNI 192.168.203.57@tcp231 [ 1519.593797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1519.646328] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 1528.610922] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:06:09 (1780337169) [ 1528.925427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.994505] LNet: Removed LNI 192.168.203.57@tcp231 [ 1529.042690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1529.139306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1529.173757] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1529.944569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1529.986500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1530.015644] LNet: Removed LNI 192.168.203.57@tcp232 [ 1530.043688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1530.078050] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1530.874533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1530.946850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1531.018940] LNet: Removed LNI 192.168.203.57@tcp232 [ 1531.037237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1531.065825] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1531.755435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1531.787960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1531.831776] LNet: Removed LNI 192.168.203.57@tcp232 [ 1531.859831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1531.891131] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 1532.591684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1532.630931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1532.672814] LNet: Removed LNI 192.168.203.57@tcp232 [ 1532.692574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1539.093675] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:06:19 (1780337179) [ 1540.171716] Key type .llcrypt unregistered [ 1540.183419] Key type ._llcrypt unregistered [ 1545.543469] Key type ._llcrypt registered [ 1545.547244] Key type .llcrypt registered [ 1545.790144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1545.817360] alg: No test for adler32 (adler32-zlib) [ 1546.789976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1546.823536] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1555.148892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1564.984911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.045307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1565.082626] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1567.266253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 2 [ 1567.320639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 2 [ 1567.379136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 2 [ 1567.462932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 2 [ 1567.534461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_interrupt -r 2 [ 1567.589924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 2 [ 1567.643096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_interrupt -r 2 [ 1567.703301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 2 [ 1567.767139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1567.788581] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1567.798687] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1568.274109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1568.319506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1568.370569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1568.423262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1568.472550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1568.524537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1568.572676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1568.652477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1568.677792] LNet: 86097:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1568.706665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1568.779563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1568.832959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1568.864264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1568.913753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1568.961181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1569.708435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 2 [ 1569.747247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 2 [ 1569.784039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 2 [ 1569.835239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 2 [ 1569.921567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_dropped -r 2 [ 1569.977281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 2 [ 1570.026125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_dropped -r 2 [ 1570.084855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 2 [ 1570.146409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1570.205685] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1570.420456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1570.448512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1570.484783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1570.514965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1570.536634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1570.559477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1570.587622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1570.610308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1570.628916] LNet: 86218:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1570.648116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1570.694515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1570.731866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1570.771921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1570.827526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1570.872283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1571.231912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 2 [ 1571.285067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 2 [ 1571.317938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 2 [ 1571.351980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 2 [ 1571.384493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_aborted -r 2 [ 1571.413339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 2 [ 1571.458782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_aborted -r 2 [ 1571.501330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 2 [ 1571.545546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1571.589355] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1571.886178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1571.933933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1571.972316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1572.028111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1572.111884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1572.177188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1572.215848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1572.270823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1572.317706] LNet: 86338:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1572.351472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1572.386212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1572.411937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1572.474746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1572.511539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1572.568352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1573.159870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 2 [ 1573.190763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 2 [ 1573.229628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 2 [ 1573.274087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 2 [ 1573.319762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_no_route -r 2 [ 1573.384650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 2 [ 1573.466729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_no_route -r 2 [ 1573.542066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 2 [ 1573.612679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1573.676141] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1574.017844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1574.070590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1574.133918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1574.163991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1574.207204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1574.259151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1574.303917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1574.342688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1574.363729] LNet: 86458:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1574.407523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1574.467598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1574.506327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1574.557484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1574.602407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1574.655852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1575.309130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 2 [ 1575.378424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 2 [ 1575.436233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 2 [ 1575.487191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 2 [ 1575.541362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_timeout -r 2 [ 1575.638061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 2 [ 1575.707312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_timeout -r 2 [ 1575.772522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 2 [ 1575.828515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1576.182483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1576.239767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1576.318702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1576.395910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1576.445900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1576.503514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1576.559762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1576.595306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1576.649074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1576.696898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1576.766561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1576.815226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1576.899262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1577.013623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1577.763891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 2 [ 1577.814464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 2 [ 1577.868715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 2 [ 1577.926282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 2 [ 1577.995669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_dropped -r 2 [ 1578.061551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 2 [ 1578.110301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_dropped -r 2 [ 1578.163770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 2 [ 1578.242415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 1578.304577] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 1578.315879] LNet: Skipped 1 previous similar message [ 1578.333042] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 1578.676634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.719258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1578.782545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.912457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.978601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1579.050295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1579.095464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1579.168617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1579.201731] LNet: 86698:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1579.218135] LNet: 86698:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1579.266499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1579.327264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1579.402567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1579.451210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1579.505786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1579.543513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1586.438814] LNet: 86774:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.460828] LNetError: 86774:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.478199] LNet: Removed LNI 192.168.203.57@tcp [ 1586.488161] LNet: Removed LNI 192.168.203.57@tcp1 [ 1587.138254] Key type .llcrypt unregistered [ 1587.141201] Key type ._llcrypt unregistered [ 1606.151457] Key type ._llcrypt registered [ 1606.164261] Key type .llcrypt registered [ 1608.374752] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 14:07:28 (1780337248) [ 1608.885781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.917192] alg: No test for adler32 (adler32-zlib) [ 1610.013449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.122725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.172062] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1611.781803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1611.890437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 1611.926459] LNetError: 87754:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1612.065821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.57@tcp [ 1612.264885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1622.061523] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 14:07:42 (1780337262) [ 1622.206879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.262509] LNet: 88336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.279370] LNetError: 88336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.326700] LNet: Removed LNI 192.168.203.57@tcp [ 1622.360737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.487081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.518246] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1623.757764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1623.829187] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1625.369177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.57@tcp [ 1625.407640] LNetError: 88610:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 1625.503478] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 1625.505127] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 1626.663140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.57@tcp [ 1635.848620] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 14:07:56 (1780337276) [ 1637.841765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg357-server [ 1640.090113] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 14:08:00 (1780337280) [ 1642.080176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg357-server [ 1644.061472] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 14:08:04 (1780337284) [ 1646.121800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg357-server [ 1648.395671] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:08:08 (1780337288) [ 1648.524326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.547082] LNet: 90227:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1648.558462] LNetError: 90227:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1648.563758] LNet: Removed LNI 192.168.203.57@tcp [ 1648.618294] LNet: Removed LNI 192.168.203.57@tcp1 [ 1648.634532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.686558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1648.721538] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1649.468360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1649.551594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1649.617147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1649.700664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1649.754335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1649.809450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1657.808434] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 14:08:18 (1780337298) [ 1657.949972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.013564] LNet: 90958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.029346] LNetError: 90958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.047876] LNetError: 90958:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1658.067264] LNet: Removed LNI 192.168.203.57@tcp [ 1658.120273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.193672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.225304] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1667.407548] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:08:27 (1780337307) [ 1667.554895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.571558] LNet: 91656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.583207] LNetError: 91656:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.604986] LNet: Removed LNI 192.168.203.57@tcp [ 1667.630967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.722562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.753239] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1675.766120] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:08:36 (1780337316) [ 1677.624297] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1679.746195] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:08:40 (1780337320) [ 1680.429429] LNet: 92698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.442392] LNetError: 92698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.459684] LNet: Removed LNI 192.168.203.57@tcp [ 1681.285662] Key type .llcrypt unregistered [ 1681.290487] Key type ._llcrypt unregistered [ 1688.229644] Key type ._llcrypt registered [ 1688.235832] Key type .llcrypt registered [ 1688.500664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1688.523471] alg: No test for adler32 (adler32-zlib) [ 1689.490492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1689.518934] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1697.794404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1703.201566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.157@tcp [ 1703.654883] LNet: 93116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.665902] LNetError: 93116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.684814] LNet: Removed LNI 192.168.203.57@tcp [ 1704.260169] Key type .llcrypt unregistered [ 1704.264925] Key type ._llcrypt unregistered [ 1716.073104] Key type ._llcrypt registered [ 1716.078800] Key type .llcrypt registered [ 1717.469997] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1719.376806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1721.604503] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:09:22 (1780337362) [ 1723.181680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1725.713494] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:09:26 (1780337366) [ 1728.854433] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg357-server [ 1731.307793] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:09:31 (1780337371) [ 1734.506982] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg357-server [ 1737.786878] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:09:37 (1780337377) [ 1739.860449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1739.874027] alg: No test for adler32 (adler32-zlib) [ 1741.034409] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1741.265031] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1743.039178] Key type lgssc registered [ 1744.389946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1768.037295] Key type lgssc unregistered [ 1768.339576] LNet: 96296:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.361770] LNetError: 96296:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.399542] LNet: Removed LNI 192.168.203.57@tcp [ 1769.278893] Key type .llcrypt unregistered [ 1769.287655] Key type ._llcrypt unregistered [ 1786.631422] Key type ._llcrypt registered [ 1786.635917] Key type .llcrypt registered [ 1788.178959] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:10:28 (1780337428) [ 1788.702279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.718986] alg: No test for adler32 (adler32-zlib) [ 1789.657747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.710154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1789.760929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1789.812662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1789.855246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1797.237586] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:10:38 (1780337438) [ 1797.327468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.351431] LNet: 97704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1797.384479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.467752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1797.503530] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1798.549970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1798.577453] LNet: 97857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.586301] LNetError: 97857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.610772] LNet: Removed LNI 192.168.203.57@tcp [ 1798.640451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.739951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1798.783517] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1799.182637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.198070] LNetError: 97885:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.211804] LNet: Removed LNI 192.168.203.57@tcp [ 1799.237581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.298354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1799.337968] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1806.180296] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:10:47 (1780337447) [ 1806.547644] LNet: 98477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.560076] LNet: 98477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1806.575067] LNetError: 98477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.608282] LNet: Removed LNI 192.168.203.57@tcp [ 1807.279610] Key type .llcrypt unregistered [ 1807.286910] Key type ._llcrypt unregistered [ 1815.139268] Key type ._llcrypt registered [ 1815.148641] Key type .llcrypt registered [ 1815.442580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1815.462843] alg: No test for adler32 (adler32-zlib) [ 1816.482429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.616681] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1816.620915] LNet: *** cfs_fail_loc=e003, val=100*** [ 1817.620074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1817.654198] LNet: *** cfs_fail_loc=e003, val=100*** [ 1817.657462] LNet: Skipped 2 previous similar messages [ 1817.662502] Ping reply received with mapping: [ 1817.665838] found metadata reply NI[192.168.203.57@tcp] [ 1817.668925] 39 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1817.672252] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1817.674800] 5a 5a 5a 5a 5a 5a 5a 5a [ 1817.936359] LNet: 98875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.962452] LNetError: 98875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.985275] LNet: Removed LNI 192.168.203.57@tcp [ 1819.676927] Key type .llcrypt unregistered [ 1819.687983] Key type ._llcrypt unregistered [ 1831.490868] Key type ._llcrypt registered [ 1831.493923] Key type .llcrypt registered [ 1832.956959] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:11:13 (1780337473) [ 1833.503336] Key type .llcrypt unregistered [ 1833.506334] Key type ._llcrypt unregistered [ 1839.200705] Key type ._llcrypt registered [ 1839.202949] Key type .llcrypt registered [ 1839.466579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1839.499462] alg: No test for adler32 (adler32-zlib) [ 1840.628515] LNetError: 99922:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1841.023574] Key type .llcrypt unregistered [ 1841.029985] Key type ._llcrypt unregistered [ 1846.070964] Key type ._llcrypt registered [ 1846.074630] Key type .llcrypt registered [ 1847.314932] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:11:28 (1780337488) [ 1847.863291] Key type .llcrypt unregistered [ 1847.865861] Key type ._llcrypt unregistered [ 1853.753954] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1853.756657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1856.389156] Key type ._llcrypt registered [ 1856.393671] Key type .llcrypt registered [ 1856.654880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1856.675335] alg: No test for adler32 (adler32-zlib) [ 1857.612410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.680477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1866.759456] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:11:47 (1780337507) [ 1866.854851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.884614] LNet: 101698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.894650] LNetError: 101698:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1866.930633] LNet: Removed LNI 10.1.2.3@tcp [ 1866.987378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.675946] Key type .llcrypt unregistered [ 1867.682726] Key type ._llcrypt unregistered [ 1873.431092] Key type ._llcrypt registered [ 1873.435087] Key type .llcrypt registered [ 1873.631824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1873.647179] alg: No test for adler32 (adler32-zlib) [ 1874.647986] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1874.714345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1874.879834] LNet: 101992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.885410] LNetError: 101992:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.900384] LNet: Removed LNI 192.168.203.57@tcp [ 1875.609544] Key type .llcrypt unregistered [ 1875.612902] Key type ._llcrypt unregistered [ 1885.883692] Key type ._llcrypt registered [ 1885.887298] Key type .llcrypt registered [ 1887.071783] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:12:08 (1780337528) [ 1887.620304] Key type .llcrypt unregistered [ 1887.623267] Key type ._llcrypt unregistered [ 1893.636859] Key type ._llcrypt registered [ 1893.642568] Key type .llcrypt registered [ 1893.899330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1893.919632] alg: No test for adler32 (adler32-zlib) [ 1901.521314] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:12:22 (1780337542) [ 1901.640291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.700072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.767237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.813376] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1902.446125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1902.478457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1902.504822] LNet: Removed LNI 192.168.203.57@tcp [ 1902.751604] LNet: 103735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.075402] Key type .llcrypt unregistered [ 1903.080756] Key type ._llcrypt unregistered [ 1914.087493] Key type ._llcrypt registered [ 1914.089524] Key type .llcrypt registered [ 1915.606832] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:12:36 (1780337556) [ 1916.013774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.032076] alg: No test for adler32 (adler32-zlib) [ 1916.993230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.068953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.128141] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1918.248274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1918.295792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.57@tcp [ 1918.321785] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1918.327307] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.333426] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 0 [ 1918.338716] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1918.342876] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1918.346977] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.350810] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 0 [ 1918.355958] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1918.359246] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1918.363133] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.367327] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 1968 [ 1918.372619] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1918.375636] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1918.379320] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.383644] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 1968 [ 1918.389343] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1925.185645] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:12:46 (1780337566) [ 1925.605871] LNet: 105275:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.614492] LNetError: 105275:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.627214] LNet: Removed LNI 192.168.203.57@tcp [ 1926.211653] Key type .llcrypt unregistered [ 1926.214900] Key type ._llcrypt unregistered [ 1931.806372] Key type ._llcrypt registered [ 1931.810324] Key type .llcrypt registered [ 1932.115216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.151351] alg: No test for adler32 (adler32-zlib) [ 1933.154872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1933.232969] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1941.369718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1950.493859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.536838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1950.561585] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1951.779905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1951.918719] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1951.921616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1951.924672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1954.453801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1954.476584] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1965.069564] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:13:26 (1780337606) [ 1965.349318] LNet: 106783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.362831] LNetError: 106783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.380032] LNet: Removed LNI 192.168.203.57@tcp [ 1966.431171] LNet: Removed LNI 192.168.203.57@tcp1 [ 1966.435377] LNetError: 106783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1966.445111] LNetError: 106783:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1966.481981] LNet: Removed LNI 10.1.2.3@tcp99 [ 1967.160598] Key type .llcrypt unregistered [ 1967.167756] Key type ._llcrypt unregistered [ 1972.965592] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1972.970503] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1975.692698] Key type ._llcrypt registered [ 1975.695222] Key type .llcrypt registered [ 1975.925087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1975.966705] alg: No test for adler32 (adler32-zlib) [ 1976.920564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.031337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.121602] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1978.182871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1978.216596] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1981.049817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1981.108312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1981.196574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1981.222819] LNetError: 107529:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1981.258408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1981.318907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1981.762304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1981.862772] LNet: 107549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.876281] LNetError: 107549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.943307] LNet: Removed LNI 192.168.203.57@tcp [ 1982.947330] LNetError: 107549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1982.970868] LNet: Removed LNI 10.1.2.3@tcp [ 1983.648739] Key type .llcrypt unregistered [ 1983.653649] Key type ._llcrypt unregistered [ 1995.392930] Key type ._llcrypt registered [ 1995.395397] Key type .llcrypt registered [ 1997.135952] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:13:57 (1780337637) [ 1997.678893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.705478] alg: No test for adler32 (adler32-zlib) [ 1998.618650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.659107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1998.681993] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1999.496968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1999.563322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg357-client@tcp [ 2009.513444] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:14:09 (1780337649) [ 2009.634713] LNet: 109078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.644619] LNetError: 109078:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.664395] LNet: Removed LNI 192.168.203.57@tcp [ 2011.467478] Key type .llcrypt unregistered [ 2011.478176] Key type ._llcrypt unregistered [ 2019.099269] Key type ._llcrypt registered [ 2019.107772] Key type .llcrypt registered [ 2019.386758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2019.445251] alg: No test for adler32 (adler32-zlib) [ 2020.437070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2020.479061] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2020.530781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2029.573490] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:14:30 (1780337670) [ 2031.379689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2033.529519] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:14:34 (1780337674) [ 2035.772721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2037.906958] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:14:38 (1780337678) [ 2038.183452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.222217] LNet: 110507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.230256] LNetError: 110507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.249089] LNet: Removed LNI 192.168.203.57@tcp [ 2038.286118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.357327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2038.408335] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2039.366407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2039.469733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2046.099680] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:14:47 (1780337687) [ 2046.261990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2046.305140] LNet: 111217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.310153] LNetError: 111217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.322486] LNet: Removed LNI 192.168.203.57@tcp [ 2046.346627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.405267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2046.438968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2047.100230] Key type .llcrypt unregistered [ 2047.103831] Key type ._llcrypt unregistered [ 2056.833495] Key type ._llcrypt registered [ 2056.837027] Key type .llcrypt registered [ 2058.191254] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:14:59 (1780337699) [ 2058.556425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2058.574944] alg: No test for adler32 (adler32-zlib) [ 2059.538252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.657288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.683740] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2060.204235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2060.268602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2066.848221] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:15:07 (1780337707) [ 2066.982617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.023939] LNet: 112772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.032566] LNetError: 112772:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.044394] LNet: Removed LNI 192.168.203.57@tcp [ 2067.067527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.130021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2067.175660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2073.576849] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 14:15:14 (1780337714) [ 2073.728408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.765405] LNet: 113348:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.844094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.954855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2074.650660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2074.714786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2074.787885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2075.303415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2075.333804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2075.445318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2075.466621] LNet: 113385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.490260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.547728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2076.002576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2076.044685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2076.118195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2076.734111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2076.781293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2083.297909] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 14:15:23 (1780337723) [ 2083.380324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.421508] LNet: 113978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.455779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.555361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2083.603534] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2084.393279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2084.419286] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2085.306772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2085.410334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2085.608698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.626651] LNetError: 114264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.687183] LNet: Removed LNI 192.168.203.57@tcp [ 2086.691365] LNetError: 114264:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.721720] LNet: Removed LNI 192.168.203.57@tcp2 [ 2086.757926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.808665] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2086.817611] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2086.827153] LNet: 114271:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2087.422738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2087.608771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.647725] LNet: 114308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.659167] LNet: 114308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2088.672859] LNet: Removed LNI 192.168.203.57@tcp [ 2088.684082] LNet: Removed LNI 192.168.203.57@tcp2 [ 2088.732481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.831606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.871807] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2089.691970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2089.716535] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2090.839827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2090.894641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2091.006940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2091.298296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.319331] LNetError: 114593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2091.329259] LNetError: 114593:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2091.337605] LNet: Removed LNI 192.168.203.57@tcp [ 2092.396608] LNet: Removed LNI 192.168.203.57@tcp2 [ 2092.469743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.544166] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2092.552480] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2092.563263] LNet: 114599:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2093.069653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2093.278872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.297719] LNet: Removed LNI 192.168.203.57@tcp [ 2094.311815] LNet: Removed LNI 192.168.203.57@tcp2 [ 2094.341795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.429541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2094.459814] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2095.227781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2095.262355] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2096.299870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2096.376971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2096.646816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2096.668282] LNet: 114916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2096.673654] LNet: 114916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2096.681452] LNetError: 114916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2096.687529] LNetError: 114916:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2096.694908] LNet: Removed LNI 192.168.203.57@tcp [ 2096.709567] LNet: Removed LNI 192.168.203.57@tcp2 [ 2096.735814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.798313] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2096.812366] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2096.821276] LNet: 114922:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2097.274854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2097.467712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.528648] LNet: Removed LNI 192.168.203.57@tcp [ 2098.551899] LNet: Removed LNI 192.168.203.57@tcp2 [ 2098.585245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.690908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2098.714993] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2099.481859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2099.509815] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2100.557718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2100.630449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2100.698746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2100.924843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.983199] LNet: Removed LNI 192.168.203.57@tcp [ 2103.018265] LNet: Removed LNI 192.168.203.57@tcp2 [ 2103.050710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.144656] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2103.152884] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2103.161378] LNet: 115250:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.651204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2103.844135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.866721] LNet: Removed LNI 192.168.203.57@tcp [ 2104.936079] LNet: Removed LNI 192.168.203.57@tcp2 [ 2104.970957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.073642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2105.138404] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2106.003712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2106.030811] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2107.022876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2107.106236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2107.198143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2107.576548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.613313] LNetError: 115584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.620166] LNetError: 115584:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2107.627114] LNet: Removed LNI 192.168.203.57@tcp [ 2107.643936] LNet: Removed LNI 192.168.203.57@tcp2 [ 2107.697257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.810696] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2107.830284] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2107.847232] LNet: 115590:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2108.418561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2108.704509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2109.791689] LNet: Removed LNI 192.168.203.57@tcp [ 2109.808220] LNet: Removed LNI 192.168.203.57@tcp2 [ 2109.858810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.953881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.979141] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2110.792663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2110.809656] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2111.706108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2111.774302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2111.825702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2111.915993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2112.380973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.439209] LNet: Removed LNI 192.168.203.57@tcp [ 2114.471623] LNet: Removed LNI 192.168.203.57@tcp2 [ 2114.504592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.584538] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2114.590716] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 2115.264274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2123.328196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2125.555417] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 14:16:06 (1780337766) [ 2125.723227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.759301] LNet: 116705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2125.784512] LNet: 116705:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2125.802143] LNetError: 116705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2125.816581] LNetError: 116705:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2125.827974] LNet: Removed LNI 192.168.203.57@tcp [ 2126.890560] LNet: Removed LNI 192.168.203.57@tcp2 [ 2126.933187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.309484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.328636] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2128.203727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2128.233357] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2129.465719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.494344] LNet: Removed LNI 192.168.203.57@tcp [ 2129.513087] LNet: Removed LNI 10.1.2.3@tcp [ 2129.581954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.640859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2129.661872] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2129.666865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2130.514829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2130.540161] LNet: Removed LNI 192.168.203.57@tcp [ 2130.555329] LNet: Removed LNI 10.1.2.3@tcp [ 2138.580362] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:16:19 (1780337779) [ 2139.005877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.123545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.406874] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2139.671221] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2139.677327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2139.684156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2142.221466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.298800] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2143.699857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2143.740292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2146.996478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 2147.021422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2147.091676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.57@tcp -m PUT -e local_timeout -r 1 [ 2147.205684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2147.344738] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2147.356231] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2147.726789] LNetError: 117649:0:(lib-move.c:4710:lnet_parse()) 192.168.203.57@tcp, src 192.168.203.57@tcp: Dropping PUT (error -108 looking up sender) [ 2149.791919] LNet: Removed LNI 192.168.203.57@tcp [ 2149.816822] LNet: Removed LNI 10.1.2.3@tcp [ 2150.420508] Key type .llcrypt unregistered [ 2150.422624] Key type ._llcrypt unregistered [ 2164.697647] Key type ._llcrypt registered [ 2164.701407] Key type .llcrypt registered [ 2165.906179] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:16:46 (1780337806) [ 2166.401408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.442331] alg: No test for adler32 (adler32-zlib) [ 2167.382391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.696672] LNet: 118961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.363360] Key type .llcrypt unregistered [ 2168.365745] Key type ._llcrypt unregistered [ 2174.887341] Key type ._llcrypt registered [ 2174.889446] Key type .llcrypt registered [ 2175.092528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2175.105827] alg: No test for adler32 (adler32-zlib) [ 2176.022176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2176.053179] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2183.211768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 2183.471979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2184.111209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.157@tcp [ 2184.288623] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 2184.358939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2184.385906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2184.412454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2184.453171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2184.489034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2184.515148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2184.568521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2184.619584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2184.640598] LNet: 119412:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2184.670542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2184.744763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2184.809626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2184.859162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2184.905781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2184.958808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2185.591990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2185.908503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2185.935464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2185.959533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2185.994969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2186.057485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2186.113753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2186.165795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2186.210911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2186.233255] LNet: 119529:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2186.257815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2186.301322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2186.340510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2186.372811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2186.435842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2186.489115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2187.085321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.157@tcp [ 2187.304974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2187.326380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2187.373778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2187.405559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2187.457698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2187.495629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2187.531315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2187.572993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2187.602641] LNet: 119646:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2187.632090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2187.683574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2187.711352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2187.746847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2187.783923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2187.829213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2188.485155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2188.762479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2188.817840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2188.856095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2188.889495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2188.920880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2188.958971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2189.010580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2189.043519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2189.092460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2189.129650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2189.162224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2189.202925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2189.232937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2189.286763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2189.662807] LNet: 119810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.678764] LNetError: 119810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.695825] LNet: Removed LNI 192.168.203.57@tcp [ 2190.077854] Key type .llcrypt unregistered [ 2190.080179] Key type ._llcrypt unregistered [ 2205.435054] Key type ._llcrypt registered [ 2205.438462] Key type .llcrypt registered [ 2206.955475] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:17:27 (1780337847) [ 2207.324737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2207.345472] alg: No test for adler32 (adler32-zlib) [ 2208.286631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.772173] LNet: 120698:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.515637] Key type .llcrypt unregistered [ 2209.519422] Key type ._llcrypt unregistered [ 2216.370052] Key type ._llcrypt registered [ 2216.373335] Key type .llcrypt registered [ 2216.614225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2216.643557] alg: No test for adler32 (adler32-zlib) [ 2217.655526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2217.692063] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2226.258550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 2235.824264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.877499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2235.909808] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 2237.138866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2238.088493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.157@tcp [ 2238.180565] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 2238.524723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2238.595612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2238.677944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2238.734417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2238.804462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2238.881109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2238.972586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2239.024495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2239.056273] LNet: 121351:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.086320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2239.156327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2239.195734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2239.253182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2239.303695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2239.345850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2240.275068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2240.536621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2240.591275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2240.633434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2240.685549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2240.777530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2240.820398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2240.858082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2240.931547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2240.961713] LNet: 121469:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2240.995310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2241.050279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2241.130926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2241.186474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2241.226861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2241.278661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2241.968697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.157@tcp [ 2242.294761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2242.334525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2242.382412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2242.448614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.494337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.542527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.625480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.670586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.698720] LNet: 121586:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.723836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2242.777083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2242.849592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2242.911208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2242.995520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2243.055422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2243.964924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2244.320813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2244.387550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2244.417744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2244.469460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2244.518400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2244.594126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2244.663425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2244.723109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2244.743814] LNet: 121704:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2244.765885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2244.814167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2244.886278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2244.931349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2244.990165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2245.049477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2251.594501] LNet: 121785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.608968] LNetError: 121785:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.620294] LNet: Removed LNI 192.168.203.57@tcp [ 2251.633460] LNet: Removed LNI 192.168.203.57@tcp1 [ 2252.217631] Key type .llcrypt unregistered [ 2252.219465] Key type ._llcrypt unregistered [ 2271.931229] Key type ._llcrypt registered [ 2271.936557] Key type .llcrypt registered [ 2273.628875] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 14:18:34 (1780337914) [ 2282.709356] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 14:18:43 (1780337923) [ 2290.354420] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 14:18:51 (1780337931) [ 2298.582505] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 14:18:59 (1780337939) [ 2306.640141] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 14:19:07 (1780337947) [ 2314.062991] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 14:19:15 (1780337955) [ 2321.228723] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 14:19:22 (1780337962) [ 2328.724865] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 14:19:29 (1780337969) [ 2335.473482] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 14:19:36 (1780337976) [ 2342.447357] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 14:19:43 (1780337983) [ 2350.172748] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 14:19:50 (1780337990) [ 2356.903793] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 14:19:58 (1780337998) [ 2362.519826] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 14:20:03 (1780338003) [ 2368.546124] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 14:20:09 (1780338009) [ 2375.271719] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 14:20:16 (1780338016) [ 2382.353612] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 14:20:23 (1780338023) [ 2389.914188] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 14:20:30 (1780338030) [ 2397.309499] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 14:20:38 (1780338038) [ 2397.801465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2397.815266] alg: No test for adler32 (adler32-zlib) [ 2398.747730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2398.940211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2398.975429] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2400.103185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.140537] LNet: 132499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.148946] LNetError: 132499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.187328] LNet: Removed LNI 192.168.203.57@tcp [ 2400.232660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.363495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2400.425775] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2407.743400] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 14:20:48 (1780338048) [ 2414.621991] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 14:20:55 (1780338055) [ 2422.020403] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 14:21:02 (1780338062) [ 2429.663685] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 14:21:10 (1780338070) [ 2437.067865] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 14:21:17 (1780338077) [ 2444.188470] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 14:21:25 (1780338085) [ 2452.098759] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 14:21:33 (1780338093) [ 2460.007656] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 14:21:41 (1780338101) [ 2467.143528] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 14:21:48 (1780338108) [ 2475.458508] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 14:21:56 (1780338116) [ 2483.567108] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 14:22:04 (1780338124) [ 2490.532440] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 14:22:11 (1780338131) [ 2498.919859] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2284 sec ========== 14:22:19 (1780338139) [ 2499.383452] LNet: 139968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.390630] LNetError: 139968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.405405] LNet: Removed LNI 192.168.203.57@tcp [ 2500.257604] Key type .llcrypt unregistered [ 2500.260938] Key type ._llcrypt unregistered [ 2513.500241] Key type ._llcrypt registered [ 2513.502980] Key type .llcrypt registered [ 2513.802719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2513.821881] alg: No test for adler32 (adler32-zlib) [ 2514.806897] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2515.114294] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 2516.791409] Key type lgssc registered [ 2517.974652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2572.669506] Lustre: Mounted lustre-client [ 2579.043512] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2589.883195] LustreError: 142432:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9a46074bb800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2589.926630] Lustre: Unmounted lustre-client [ 2619.102981] Key type lgssc unregistered [ 2619.457812] LNet: 142914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.465057] LNetError: 142914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.484447] LNet: Removed LNI 192.168.203.57@tcp [ 2620.447709] Key type .llcrypt unregistered [ 2620.453860] Key type ._llcrypt unregistered