[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443331268 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002371] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007770] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011054] Yama: becoming mindful. [ 0.012034] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.021757] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026337] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027124] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030044] *** VALIDATE tmpfs *** [ 0.031428] *** VALIDATE proc *** [ 0.032216] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.035170] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036132] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039006] Speculative Store Bypass: Vulnerable [ 0.042030] debug: unmapping init [mem 0xffffffff89c59000-0xffffffff89c60fff] [ 0.044795] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045599] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047006] ... bit width: 48 [ 0.047985] ... generic registers: 4 [ 0.048011] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050009] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.053199] rcu: Hierarchical SRCU implementation. [ 0.055275] smp: Bringing up secondary CPUs ... [ 0.056547] x86: Booting SMP configuration: [ 0.057020] .... node #0, CPUs: #1 #2 #3 [ 0.065463] smp: Brought up 1 node, 4 CPUs [ 0.067011] smpboot: Max logical packages: 1 [ 0.068022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143349] node 0 deferred pages initialised in 73ms [ 0.148008] devtmpfs: initialized [ 0.149212] x86/mm: Memory block size: 128MB [ 0.151791] gcov: version magic: 0x41383552 [ 0.153286] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.154074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155306] pinctrl core: initialized pinctrl subsystem [ 0.156256] [ 0.156775] ************************************************************* [ 0.157019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.158013] ** ** [ 0.159015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160013] ** ** [ 0.161011] ** This means that this kernel is built to expose internal ** [ 0.162015] ** IOMMU data structures, which may compromise security on ** [ 0.163014] ** your system. ** [ 0.164015] ** ** [ 0.165012] ** If you see this message and you are not debugging the ** [ 0.166014] ** kernel, report this immediately to your vendor! ** [ 0.167013] ** ** [ 0.168014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169011] ************************************************************* [ 0.170628] NET: Registered protocol family 16 [ 0.171498] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.173053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175032] cpuidle: using governor menu [ 0.177033] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180512] PCI: Using configuration type 1 for base access [ 0.182164] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.192111] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.193024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.195046] cryptd: max_cpu_qlen set to 1000 [ 0.198270] ACPI: Added _OSI(Module Device) [ 0.199013] ACPI: Added _OSI(Processor Device) [ 0.200012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201077] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205334] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.207579] ACPI: Interpreter enabled [ 0.208065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.209012] ACPI: Using IOAPIC for interrupt routing [ 0.210115] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.211471] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222008] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.225079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227398] acpiphp: Slot [2] registered [ 0.228180] acpiphp: Slot [5] registered [ 0.229187] acpiphp: Slot [6] registered [ 0.230115] acpiphp: Slot [3] registered [ 0.231103] acpiphp: Slot [4] registered [ 0.232094] acpiphp: Slot [7] registered [ 0.233133] acpiphp: Slot [8] registered [ 0.234143] acpiphp: Slot [9] registered [ 0.235105] acpiphp: Slot [10] registered [ 0.236101] acpiphp: Slot [11] registered [ 0.237093] acpiphp: Slot [12] registered [ 0.238102] acpiphp: Slot [13] registered [ 0.239080] acpiphp: Slot [14] registered [ 0.240070] acpiphp: Slot [15] registered [ 0.241085] acpiphp: Slot [16] registered [ 0.242095] acpiphp: Slot [17] registered [ 0.243082] acpiphp: Slot [18] registered [ 0.244126] acpiphp: Slot [19] registered [ 0.245114] acpiphp: Slot [20] registered [ 0.246108] acpiphp: Slot [21] registered [ 0.247126] acpiphp: Slot [22] registered [ 0.248104] acpiphp: Slot [23] registered [ 0.249108] acpiphp: Slot [24] registered [ 0.250108] acpiphp: Slot [25] registered [ 0.251120] acpiphp: Slot [26] registered [ 0.252118] acpiphp: Slot [27] registered [ 0.253139] acpiphp: Slot [28] registered [ 0.254140] acpiphp: Slot [29] registered [ 0.255108] acpiphp: Slot [30] registered [ 0.256115] acpiphp: Slot [31] registered [ 0.257109] PCI host bridge to bus 0000:00 [ 0.258027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.262022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.266024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.269030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.274072] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.277044] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.279232] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.283178] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.287000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.300845] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306024] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.308015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.311017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313629] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.315690] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.318039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.320859] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.325000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.334016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.337000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.337000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.344102] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.352015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.369014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.384000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.392017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.403014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.447034] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.449353] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.451364] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.454474] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.457237] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.462077] iommu: Default domain type: Passthrough [ 0.463360] SCSI subsystem initialized [ 0.465207] ACPI: bus type USB registered [ 0.467114] usbcore: registered new interface driver usbfs [ 0.469172] usbcore: registered new interface driver hub [ 0.471079] usbcore: registered new device driver usb [ 0.473159] pps_core: LinuxPPS API ver. 1 registered [ 0.475012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.479073] PTP clock support registered [ 0.481089] EDAC MC: Ver: 3.0.0 [ 0.482474] PCI: Using ACPI for IRQ routing [ 0.484805] NetLabel: Initializing [ 0.486016] NetLabel: domain hash size = 128 [ 0.488010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.490080] NetLabel: unlabeled traffic allowed by default [ 0.492139] vgaarb: loaded [ 0.494321] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.496017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.504040] clocksource: Switched to clocksource kvm-clock [ 0.625302] VFS: Disk quotas dquot_6.6.0 [ 0.627173] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.630192] *** VALIDATE ramfs *** [ 0.631769] *** VALIDATE hugetlbfs *** [ 0.633736] pnp: PnP ACPI init [ 0.636429] pnp: PnP ACPI: found 6 devices [ 0.654254] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.658018] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.660569] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.663304] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.666411] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.669133] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.672437] NET: Registered protocol family 2 [ 0.675511] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.681307] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.685490] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.691430] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.695506] TCP: Hash tables configured (established 65536 bind 65536) [ 0.698827] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.701987] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705257] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708490] NET: Registered protocol family 1 [ 0.711621] RPC: Registered named UNIX socket transport module. [ 0.713118] RPC: Registered udp transport module. [ 0.714477] RPC: Registered tcp transport module. [ 0.715783] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.717654] NET: Registered protocol family 44 [ 0.718881] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.720532] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.722202] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.723951] PCI: CLS 0 bytes, default 64 [ 0.726141] Unpacking initramfs... [ 2.188129] debug: unmapping init [mem 0xffff9b2dbcc64000-0xffff9b2dbffcffff] [ 2.193629] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.196066] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.199387] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.674650] Initialise system trusted keyrings [ 2.676373] Key type blacklist registered [ 2.678760] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.697566] zbud: loaded [ 2.704377] *** VALIDATE nfs *** [ 2.717903] *** VALIDATE nfs4 *** [ 2.720210] pstore: using deflate compression [ 2.724469] Platform Keyring initialized [ 2.833669] NET: Registered protocol family 38 [ 2.835566] Key type asymmetric registered [ 2.837382] Asymmetric key parser 'x509' registered [ 2.839296] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.842669] io scheduler mq-deadline registered [ 2.844659] io scheduler kyber registered [ 2.846693] io scheduler bfq registered [ 2.848955] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.852616] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.855844] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.858665] ACPI: Power Button [PWRF] [ 2.863589] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.870405] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.900723] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.930179] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.962019] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.966947] Non-volatile memory driver v1.3 [ 2.968903] Linux agpgart interface v0.103 [ 2.997787] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.001124] vda: detected capacity change from 0 to 76681216 [ 3.024285] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.027644] vdb: detected capacity change from 0 to 1073741824 [ 3.041896] libphy: Fixed MDIO Bus: probed [ 3.047397] usbcore: registered new interface driver usbserial_generic [ 3.050423] usbserial: USB Serial support registered for generic [ 3.052663] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.057937] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.060036] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.062642] mousedev: PS/2 mouse device common for all mice [ 3.065632] rtc_cmos 00:05: RTC can wake from S4 [ 3.068804] rtc_cmos 00:05: registered as rtc0 [ 3.070912] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.073751] intel_pstate: CPU model not supported [ 3.078162] hid: raw HID events driver (C) Jiri Kosina [ 3.081358] usbcore: registered new interface driver usbhid [ 3.081930] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.084666] usbhid: USB HID core driver [ 3.090028] drop_monitor: Initializing network drop monitor service [ 3.092576] Initializing XFRM netlink socket [ 3.093496] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input2 [ 3.097429] NET: Registered protocol family 10 [ 3.103495] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input4 [ 3.104378] Segment Routing with IPv6 [ 3.108366] NET: Registered protocol family 17 [ 3.109910] mpls_gso: MPLS GSO support [ 3.114495] RAS: Correctable Errors collector initialized. [ 3.150485] AVX version of gcm_enc/dec engaged. [ 3.153640] AES CTR mode by8 optimization enabled [ 3.258546] sched_clock: Marking stable (3258466211, 0)->(4236672777, -978206566) [ 3.262784] registered taskstats version 1 [ 3.266295] Loading compiled-in X.509 certificates [ 3.268657] zswap: loaded using pool lzo/zbud [ 3.306479] Key type big_key registered [ 3.323372] Key type encrypted registered [ 3.324905] ima: No TPM chip found, activating TPM-bypass! [ 3.327613] ima: Allocated hash algorithm: sha1 [ 3.330502] ima: No architecture policies found [ 3.332769] evm: Initialising EVM extended attributes: [ 3.335490] evm: security.selinux [ 3.336716] evm: security.ima [ 3.337820] evm: security.capability [ 3.339242] evm: HMAC attrs: 0x1 [ 3.341693] rtc_cmos 00:05: setting system clock to 2026-09-03 19:15:58 UTC (1788462958) [ 3.347120] debug: unmapping init [mem 0xffffffff8ac03000-0xffffffff8adfffff] [ 3.350351] debug: unmapping init [mem 0xffffffff89982000-0xffffffff89c58fff] [ 3.355054] Write protecting the kernel read-only data: 28672k [ 3.358913] debug: unmapping init [mem 0xffffffff88003000-0xffffffff881fffff] [ 3.361892] debug: unmapping init [mem 0xffffffff88914000-0xffffffff889fffff] [ 3.395211] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.404984] systemd[1]: Detected virtualization kvm. [ 3.408401] systemd[1]: Detected architecture x86-64. [ 3.410636] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.442860] systemd[1]: No hostname configured. [ 3.449409] systemd[1]: Set hostname to . [ 3.453680] random: systemd: uninitialized urandom read (16 bytes read) [ 3.456172] systemd[1]: Initializing machine ID from random generator. [ 3.508129] random: ln: uninitialized urandom read (6 bytes read) [ 3.644494] random: systemd: uninitialized urandom read (16 bytes read) [ 3.648500] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.663331] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ 3.682978] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Timers. Starting Journal Service... [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.758637] device-mapper: uevent: version 1.0.3 [ 4.762291] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 6.129279] virtio_net virtio0 ens2: renamed from eth0 Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 6.631868] scsi host0: ata_piix [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.690380] scsi host1: ata_piix [ 6.691916] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.693985] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 6.784776] random: fast init done [ 11.847316] random: crng init done [ 11.848486] random: 7 urandom warning(s) missed due to ratelimiting [ 14.719314] dracut-initqueue[581]: RTNETLINK answers: File exists [ 15.692717] hrtimer: interrupt took 3021518 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 18.120938] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 25.082146] printk: systemd: 26 output lines suppressed due to ratelimiting [ 26.317557] SELinux: Disabled at runtime. [ 26.469214] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 26.487076] systemd[1]: Detected virtualization kvm. [ 26.488562] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.555534] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.575551] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.601182] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.609793] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.615166] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 28.642409] systemd[1]: Starting Journal Service... Starting Journal Service... [ 28.666051] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ 28.944574] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 31.848681] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.161017] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 34.536546] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 35.436472] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.934927] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (13s / no limit) [ **] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit)[ 43.272614] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (15s / no limit) [ *** ] A start job is running for Configur…only root support (15s / no limit)[ 44.274033] NFS: Registering the id_resolver key type [ 44.276040] Key type id_resolver registered [ 44.277520] Key type id_legacy registered [*** ] A start job is running for Configur…only root support (16s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg333-client login: [ 120.632746] libcfs: loading out-of-tree module taints kernel. [ 121.081424] Key type ._llcrypt registered [ 121.115177] Key type .llcrypt registered [ 121.680408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 121.695948] alg: No test for adler32 (adler32-zlib) [ 123.718203] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 124.906433] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 126.769570] Key type lgssc registered [ 128.720121] Lustre: Echo OBD driver; http://www.lustre.org/ [ 265.282603] Lustre: Mounted lustre-client - version 2.17.58_2_gbb63b80 [ 269.941929] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 282.228761] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing check_logdir /tmp/testlogs/ [ 287.604233] Lustre: DEBUG MARKER: oleg333-client.virtnet: executing yml_node [ 290.784863] Lustre: lustre-OST0000-osc-ffff9b2e02bf4000: disconnect after 23s idle [ 292.038990] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 294.981782] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 298.083721] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 299.933296] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 3 15:20:53 EDT 2026 [ 317.087298] Lustre: Unmounted lustre-client [ 346.435892] Key type lgssc unregistered [ 346.733968] LNet: 8409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.746974] LNetError: 8409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.765596] LNet: Removed LNI 192.168.203.33@tcp [ 347.551169] Key type .llcrypt unregistered [ 347.555172] Key type ._llcrypt unregistered [ 375.184357] Key type ._llcrypt registered [ 375.189931] Key type .llcrypt registered [ 375.462671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 375.502178] alg: No test for adler32 (adler32-zlib) [ 376.527295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 376.617720] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 376.664537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 380.659768] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 380.944801] LNet: 9291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 380.955113] LNetError: 9291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 380.973509] LNet: Removed LNI 192.168.203.33@tcp [ 381.686573] Key type .llcrypt unregistered [ 381.695193] Key type ._llcrypt unregistered [ 388.956479] Key type ._llcrypt registered [ 388.960310] Key type .llcrypt registered [ 390.619668] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 15:22:23 (1788463343) [ 391.211140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 391.235326] alg: No test for adler32 (adler32-zlib) [ 392.137822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.258807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.285296] LNet: 9699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.309800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.386989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.404988] LNet: 9706:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 400.726698] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 15:22:34 (1788463354) [ 401.092394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.118857] LNet: 10284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.156689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.228952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 409.092598] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 15:22:42 (1788463362) [ 409.449823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.481378] LNet: 10871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.500199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.575891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 417.010332] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 15:22:50 (1788463370) [ 417.348145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.365137] LNet: 11456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.382867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.494976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 424.995026] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 15:22:58 (1788463378) [ 425.135741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.183510] LNet: 12035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.238277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.412408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 426.193971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 434.000746] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 15:23:07 (1788463387) [ 434.160022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.213160] LNet: 12632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.245180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.351636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 434.817993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 435.252809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 442.517738] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 15:23:16 (1788463396) [ 442.729133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.784649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.862620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 449.758175] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 15:23:23 (1788463403) [ 449.869956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.940339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.072038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 450.097863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 450.391804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 450.419274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 450.786390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 450.837025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 451.677454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 451.745462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 452.301833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 452.361915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 452.976413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 453.028813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 453.638741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 453.685289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 454.384983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 454.447663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 454.931611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 454.961922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 455.382772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 455.423577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 455.844833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 455.884165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 456.308284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 456.352675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 456.723864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 456.796849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 464.086685] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 15:23:37 (1788463417) [ 464.217345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.256155] LNet: 14490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.260281] LNet: 14490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 464.278425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.385464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 464.446870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 471.416436] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 15:23:45 (1788463425) [ 471.971767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.053922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.232581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 472.306650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 480.267254] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 15:23:53 (1788463433) [ 480.571568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.719791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.819483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 480.867465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 487.991582] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 15:24:01 (1788463441) [ 488.132578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.220254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.337165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 488.389821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 495.489394] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 15:24:09 (1788463449) [ 495.757796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.804518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.871960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 495.909513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 502.871321] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 15:24:16 (1788463456) [ 502.961968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.989109] LNet: 17422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.992108] LNet: 17422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 503.016565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.090174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 503.126845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 510.490831] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 15:24:24 (1788463464) [ 510.722933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.754987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.845100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.178963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 518.717864] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 15:24:32 (1788463472) [ 519.311879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.371502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.544318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 519.705452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.175980] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 15:24:41 (1788463481) [ 527.612970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.680624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.880561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.089277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 536.698308] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 15:24:50 (1788463490) [ 536.932261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.966979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.548487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 537.787305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 545.005854] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 15:24:58 (1788463498) [ 545.177958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.242487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.316645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.366672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 552.383981] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 15:25:06 (1788463506) [ 552.855110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.952675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.048880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 553.099642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 562.588628] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 15:25:15 (1788463515) [ 562.765763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.795335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.848693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 570.012525] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 15:25:23 (1788463523) [ 570.349261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.366304] LNet: 22282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.370707] LNet: 22282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 570.384568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.439367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 577.927776] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 15:25:31 (1788463531) [ 578.066337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 578.117919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 578.255302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 578.629632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 586.469696] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 15:25:40 (1788463540) [ 586.700674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 586.742816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.842878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 586.880459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 594.823978] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 15:25:48 (1788463548) [ 595.193256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.257322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.402548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 595.434962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 603.164511] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 15:25:57 (1788463557) [ 603.261305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.317873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.452914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 603.500937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 610.442452] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 15:26:04 (1788463564) [ 610.646926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.705626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.818944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 610.871635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 617.383523] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 15:26:11 (1788463571) [ 617.615537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.664074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.733773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 617.816305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 617.835283] LNetError: 25814:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 617.869322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 617.902434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 624.253695] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 15:26:18 (1788463578) [ 624.464265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 624.515945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 624.652278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q9ZvvQPThR [ 624.747299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q9ZvvQPThR [ 624.832620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Q9ZvvQPThR [ 631.413747] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 15:26:25 (1788463585) [ 632.037838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.077969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.129882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 632.185557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 632.269079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 639.757875] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 15:26:33 (1788463593) [ 639.919559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.023981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.220635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 640.263239] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 642.350907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 644.375643] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 646.264154] Key type lgssc registered [ 647.293076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 652.261118] LNetError: 27573:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.33@tcp99 has no available nets [ 655.848864] Key type lgssc unregistered [ 656.334895] LNetError: 28682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.387642] LNet: Removed LNI 192.168.203.33@tcp [ 658.312236] Key type .llcrypt unregistered [ 658.320784] Key type ._llcrypt unregistered [ 665.600537] Key type ._llcrypt registered [ 665.602919] Key type .llcrypt registered [ 666.731208] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 15:27:00 (1788463620) [ 667.537346] Key type .llcrypt unregistered [ 667.545664] Key type ._llcrypt unregistered [ 674.336157] Key type ._llcrypt registered [ 674.338366] Key type .llcrypt registered [ 674.602070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 674.626212] alg: No test for adler32 (adler32-zlib) [ 675.617546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.696757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 675.720634] LNet: 29591:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 675.748774] LNet: 29591:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 675.759625] LNet: 29591:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 675.792666] LNet: Message forwarding will be enabled in 2s [ 677.856293] LNet: Message forwarding enabled [ 678.280943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 678.300978] LNet: Message forwarding will stop in 2s to 6s [ 681.954566] LNet: Message forwarding disabled [ 682.857620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 682.879206] LNet: 29680:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 682.885520] LNet: 29680:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 682.895374] LNet: 29680:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 682.908909] LNet: Message forwarding will be enabled in 2s [ 683.101421] LNet: 29693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.756668] Key type .llcrypt unregistered [ 683.763133] Key type ._llcrypt unregistered [ 697.093828] Key type ._llcrypt registered [ 697.098862] Key type .llcrypt registered [ 698.266685] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 15:27:32 (1788463652) [ 699.022987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 699.043399] alg: No test for adler32 (adler32-zlib) [ 699.965474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.059534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 706.949766] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 15:27:40 (1788463660) [ 707.056109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.083583] LNet: 31087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.121425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.197660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 707.238220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 707.277610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 707.313526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 707.367314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 707.432982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 707.489483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 707.546768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 707.597784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 707.691193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 707.756801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 707.813656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 707.876495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 707.931558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 707.978629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 708.035528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 708.086966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 708.146284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 708.181421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 708.228423] LNetError: 31137:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 714.772863] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 15:27:48 (1788463668) [ 715.049703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.065221] LNet: 31708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.082381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.155950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 721.769270] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 15:27:55 (1788463675) [ 724.047850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg333-server [ 725.884633] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 15:27:59 (1788463679) [ 726.058171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.104092] LNet: 32639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.145437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.252514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 726.292383] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 727.096665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[34-38]@tcp [ 727.124691] LNet: 32773:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 727.182944] LNetError: 32644:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 727.212624] LNetError: 32644:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.34@tcp has gone from up to down [ 733.654959] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 15:28:07 (1788463687) [ 735.083656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg333-server [ 736.921549] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 15:28:10 (1788463690) [ 737.176335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.205534] LNet: 33709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.216957] LNetError: 33709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.245100] LNet: Removed LNI 192.168.203.33@tcp [ 737.294511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.390849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.431196] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 738.229404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[34-39/2]@tcp [ 738.269183] LNet: 33826:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 738.281276] LNet: 33826:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 738.300141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[34-39/2]@tcp [ 738.317673] LNetError: 33713:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 738.340754] LNetError: 33713:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 738.348917] LNetError: 33713:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.34@tcp has gone from up to down [ 738.357415] LNetError: 33713:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 744.650305] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 15:28:18 (1788463698) [ 744.919290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.949653] LNet: 34418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.961537] LNetError: 34418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.990222] LNet: Removed LNI 192.168.203.33@tcp [ 745.010326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.085473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 745.122167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.160662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.228294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 745.288770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.326204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.380332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 745.448707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.494921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.578897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 745.651587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.692917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.741865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 745.845681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.912304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 745.978829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.040375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.079094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.116384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 746.212137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.250248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.289648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.375772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.416097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.453914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 746.545306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.603087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.681860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.758819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.786235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.833333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 746.882635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.930228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.034296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.927639] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 15:28:26 (1788463706) [ 755.435344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg333-server [ 757.113230] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 15:28:31 (1788463711) [ 759.827359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg333-server [ 761.960171] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 15:28:35 (1788463715) [ 762.478240] LNet: 35821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.491765] LNet: 35821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 763.397381] Key type .llcrypt unregistered [ 763.399557] Key type ._llcrypt unregistered [ 772.652933] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 772.845181] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 772.855360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 772.863455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 775.476953] Key type ._llcrypt registered [ 775.479962] Key type .llcrypt registered [ 775.745849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.785624] alg: No test for adler32 (adler32-zlib) [ 776.791112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.873707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.905618] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 777.663749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 777.693909] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 780.062220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 780.089691] LNet: Removed LNI 10.1.2.3@tcp [ 780.256614] LNet: 36535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.262341] LNetError: 36535:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.278240] LNet: Removed LNI 192.168.203.33@tcp [ 780.918627] Key type .llcrypt unregistered [ 780.921558] Key type ._llcrypt unregistered [ 794.016735] Key type ._llcrypt registered [ 794.018587] Key type .llcrypt registered [ 795.152086] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 15:29:09 (1788463749) [ 795.891343] Key type .llcrypt unregistered [ 795.898729] Key type ._llcrypt unregistered [ 802.371054] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 802.462856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 802.472745] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 802.480824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 805.181296] Key type ._llcrypt registered [ 805.183629] Key type .llcrypt registered [ 805.386446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 805.399395] alg: No test for adler32 (adler32-zlib) [ 806.313782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.402582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 806.445527] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 807.078054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 807.120140] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.620173] LNet: 38022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.629111] LNetError: 38022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 809.633912] LNet: Removed LNI 10.1.2.3@tcp [ 809.644797] LNet: Removed LNI 192.168.203.33@tcp [ 810.256603] Key type .llcrypt unregistered [ 810.258445] Key type ._llcrypt unregistered [ 823.709835] Key type ._llcrypt registered [ 823.711562] Key type .llcrypt registered [ 825.328865] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 15:29:38 (1788463778) [ 825.937342] Key type .llcrypt unregistered [ 825.940941] Key type ._llcrypt unregistered [ 833.011047] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 833.342897] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 833.347200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 833.350878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.055265] Key type ._llcrypt registered [ 835.057759] Key type .llcrypt registered [ 835.230970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.240931] alg: No test for adler32 (adler32-zlib) [ 836.180762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.238628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 836.289077] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 838.133294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 838.178066] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.811973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 839.846121] LNet: Removed LNI 10.1.2.3@tcp [ 839.883299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 839.906244] LNet: Removed LNI 10.1.2.31@tcp [ 839.923198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 839.947382] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.620922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 840.639439] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 841.318979] LNet: 39928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.324423] LNetError: 39928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 841.331251] LNet: Removed LNI 10.1.2.31@tcp [ 841.345840] LNet: Removed LNI 10.1.2.3@tcp [ 841.826544] Key type .llcrypt unregistered [ 841.828279] Key type ._llcrypt unregistered [ 854.296886] Key type ._llcrypt registered [ 854.301541] Key type .llcrypt registered [ 855.438772] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 15:30:09 (1788463809) [ 856.232310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.265860] alg: No test for adler32 (adler32-zlib) [ 857.268725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.349323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 857.365937] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 858.186749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.203749] LNet: 40878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.210622] LNetError: 40878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 858.231696] LNet: Removed LNI 192.168.203.33@tcp [ 858.257901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.357308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.33@tcp [ 858.384476] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 859.208199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.33@tcp [ 859.253306] LNet: Removed LNI 192.168.203.33@tcp [ 859.350833] LNet: 41013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.042400] Key type .llcrypt unregistered [ 860.044989] Key type ._llcrypt unregistered [ 872.161897] Key type ._llcrypt registered [ 872.169701] Key type .llcrypt registered [ 873.255406] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 15:30:27 (1788463827) [ 875.267525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg333-server [ 877.216746] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 15:30:30 (1788463830) [ 877.790886] Key type .llcrypt unregistered [ 877.793342] Key type ._llcrypt unregistered [ 883.812576] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 883.960229] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 883.971034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 883.993523] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 886.605183] Key type ._llcrypt registered [ 886.609562] Key type .llcrypt registered [ 886.844990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.866853] alg: No test for adler32 (adler32-zlib) [ 887.819162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.892263] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 888.603258] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 890.550971] LNet: 42845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.563597] LNetError: 42845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.571076] LNet: Removed LNI 192.168.203.33@tcp1 [ 890.583673] LNet: Removed LNI 10.1.2.3@tcp2 [ 891.148533] Key type .llcrypt unregistered [ 891.150785] Key type ._llcrypt unregistered [ 901.735695] Key type ._llcrypt registered [ 901.738282] Key type .llcrypt registered [ 903.011762] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 15:30:56 (1788463856) [ 903.466154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.484500] alg: No test for adler32 (adler32-zlib) [ 904.443480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.520904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.574816] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 905.383159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 905.418451] LNet: 43814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.449163] LNetError: 43814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.478882] LNet: Removed LNI 192.168.203.33@tcp [ 905.502708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.591550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 905.641492] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 912.680258] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 15:31:06 (1788463866) [ 912.961870] LNet: 44406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.975629] LNetError: 44406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.987891] LNet: Removed LNI 192.168.203.33@tcp [ 913.667517] Key type .llcrypt unregistered [ 913.675208] Key type ._llcrypt unregistered [ 919.861185] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 920.086627] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 920.090693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 920.109882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 922.579792] Key type ._llcrypt registered [ 922.583315] Key type .llcrypt registered [ 922.780332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.797185] alg: No test for adler32 (adler32-zlib) [ 923.709796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 923.773202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 923.803334] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 924.602439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 924.634006] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.001341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.029949] LNet: 45143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.043443] LNetError: 45143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.049249] LNet: Removed LNI 10.1.2.3@tcp [ 927.060776] LNet: Removed LNI 192.168.203.33@tcp [ 927.096231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.192815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 927.238889] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 927.249839] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 927.894965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.917373] LNet: 45170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.926739] LNetError: 45170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 927.937173] LNetError: 45170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 927.946517] LNet: Removed LNI 10.1.2.3@tcp [ 927.958061] LNet: Removed LNI 192.168.203.33@tcp [ 927.990427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.052243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 928.084780] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 928.088150] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 928.681077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 928.710138] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 929.684456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.701946] LNet: 45332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.710844] LNetError: 45332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 929.722215] LNetError: 45332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 929.736491] LNet: Removed LNI 10.1.2.3@tcp [ 929.739323] LNet: Removed LNI 192.168.203.33@tcp [ 929.753766] LNet: Removed LNI 192.168.203.33@tcp2 [ 929.807948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.884261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 929.916189] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 929.925142] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 929.933441] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 937.593838] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 15:31:31 (1788463891) [ 937.805987] LNet: 45928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.837989] LNetError: 45928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 937.850389] LNetError: 45928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 937.864425] LNet: Removed LNI 10.1.2.3@tcp [ 937.870562] LNet: Removed LNI 192.168.203.33@tcp [ 937.885500] LNet: Removed LNI 192.168.203.33@tcp2 [ 938.752246] Key type .llcrypt unregistered [ 938.762695] Key type ._llcrypt unregistered [ 945.576861] Key type ._llcrypt registered [ 945.579719] Key type .llcrypt registered [ 945.727602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.752471] alg: No test for adler32 (adler32-zlib) [ 946.682114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.740450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.767215] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 947.545380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.569309] LNet: 46321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.576715] LNetError: 46321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.590702] LNet: Removed LNI 192.168.203.33@tcp [ 947.613277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.658878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 947.680705] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 948.042976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.068851] LNet: Removed LNI 192.168.203.33@tcp [ 948.086927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.143880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 948.164157] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 948.627653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.647161] LNet: 46382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.654185] LNet: 46382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 948.668149] LNetError: 46382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.675160] LNetError: 46382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 948.688841] LNet: Removed LNI 192.168.203.33@tcp [ 948.720106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.784052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 948.808605] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 949.233381] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.345703] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.351719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.355707] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.789955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 951.812918] LNet: 46456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.819491] LNetError: 46456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 951.833920] LNet: Removed LNI 192.168.203.33@tcp [ 951.851118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.944932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 951.972607] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 958.835525] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 15:31:52 (1788463912) [ 959.199636] LNet: 47055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.212348] LNetError: 47055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.235253] LNet: Removed LNI 192.168.203.33@tcp [ 959.958349] Key type .llcrypt unregistered [ 959.961364] Key type ._llcrypt unregistered [ 966.297501] Key type ._llcrypt registered [ 966.300028] Key type .llcrypt registered [ 966.441935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.456886] alg: No test for adler32 (adler32-zlib) [ 967.388552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.443414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 967.473089] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 968.190175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 968.222771] LNet: 47450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.228564] LNetError: 47450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.242419] LNet: Removed LNI 192.168.203.33@tcp [ 968.260900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.315630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 968.346749] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 975.057082] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 15:32:08 (1788463928) [ 975.384242] LNet: 48042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 975.398621] LNetError: 48042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 975.426407] LNet: Removed LNI 192.168.203.33@tcp [ 976.103528] Key type .llcrypt unregistered [ 976.108864] Key type ._llcrypt unregistered [ 982.097949] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.227865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 982.240060] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 982.253042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.703006] Key type ._llcrypt registered [ 984.705222] Key type .llcrypt registered [ 984.900476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.916521] alg: No test for adler32 (adler32-zlib) [ 985.818069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.883538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.926556] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 986.828937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 986.856562] LNet: 48490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.862157] LNetError: 48490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.883086] LNet: Removed LNI 192.168.203.33@tcp [ 986.929384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 986.989214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 987.037428] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 993.615522] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 15:32:27 (1788463947) [ 993.945996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 993.961254] LNet: 49089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.966864] LNetError: 49089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.981370] LNet: Removed LNI 192.168.203.33@tcp [ 994.010417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.110659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 999.849270] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 15:32:33 (1788463953) [ 999.988899] LNet: 49663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.837616] Key type .llcrypt unregistered [ 1000.843054] Key type ._llcrypt unregistered [ 1007.799676] Key type ._llcrypt registered [ 1007.802359] Key type .llcrypt registered [ 1008.079407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.095983] alg: No test for adler32 (adler32-zlib) [ 1009.193845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.281773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1009.319439] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1010.558711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.598945] LNet: 50065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.606610] LNetError: 50065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.629712] LNet: Removed LNI 192.168.203.33@tcp [ 1010.664264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.764363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1010.818250] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1019.820368] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 15:32:52 (1788463972) [ 1019.973742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1020.012914] LNet: 50652:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.027142] LNetError: 50652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.064193] LNet: Removed LNI 192.168.203.33@tcp [ 1020.123719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.255945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1028.051286] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 15:33:01 (1788463981) [ 1029.632459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1031.562373] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 15:33:05 (1788463985) [ 1033.649541] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1035.279486] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 15:33:09 (1788463989) [ 1036.844844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1038.332140] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 15:33:12 (1788463992) [ 1040.034121] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1041.857679] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 15:33:15 (1788463995) [ 1043.700560] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1045.998466] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 15:33:19 (1788463999) [ 1046.148641] LNet: 52707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.040540] Key type .llcrypt unregistered [ 1047.046071] Key type ._llcrypt unregistered [ 1053.984544] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1054.125407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1054.135703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.143388] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1056.619130] Key type ._llcrypt registered [ 1056.620564] Key type .llcrypt registered [ 1056.836818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1056.850869] alg: No test for adler32 (adler32-zlib) [ 1057.793577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.892914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1057.922501] LNet: Added LNI 192.168.203.33@tcp [8/512/0/180] [ 1057.933446] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1057.940683] LNet: 53025:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1058.064613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.092742] LNet: 53044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.098642] LNetError: 53044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.105473] LNet: Removed LNI 10.1.2.3@tcp [ 1058.107169] LNetError: 53044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.125681] LNet: Removed LNI 192.168.203.33@tcp [ 1058.147447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.198152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.215545] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1058.265894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.280769] LNet: 53068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.285873] LNetError: 53068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.295548] LNet: Removed LNI 192.168.203.33@tcp [ 1058.313257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.359471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.379480] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1058.381898] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1058.397634] LNet: 53076:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1058.503860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.521303] LNet: 53094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.527929] LNetError: 53094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.534203] LNet: Removed LNI 10.1.2.3@tcp [ 1058.537078] LNetError: 53094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.549301] LNet: Removed LNI 192.168.203.33@tcp [ 1058.570511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.626725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1058.655994] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1058.664902] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1058.672533] LNet: 53102:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1058.837679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.868664] LNet: 53128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.874257] LNetError: 53128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.888471] LNet: Removed LNI 10.1.2.3@tcp [ 1058.895038] LNetError: 53128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.913399] LNet: Removed LNI 192.168.203.33@tcp [ 1058.945874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.996716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.024271] LNet: Added LNI 192.168.203.33@tcp [8/256/0/182] [ 1059.028924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1059.150285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1059.188467] LNet: 53156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1059.197659] LNetError: 53156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1059.206497] LNet: Removed LNI 10.1.2.3@tcp [ 1059.211338] LNetError: 53156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.225764] LNet: Removed LNI 192.168.203.33@tcp [ 1059.237937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1059.300394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1059.335218] LNet: Added LNI 192.168.203.33@tcp [8/256/0/182] [ 1059.350038] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1059.359520] LNet: 53164:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1059.449213] LNet: Removed LNI 10.1.2.3@tcp [ 1059.464719] LNet: Removed LNI 192.168.203.33@tcp [ 1060.009795] Key type .llcrypt unregistered [ 1060.013996] Key type ._llcrypt unregistered [ 1072.483946] Key type ._llcrypt registered [ 1072.487116] Key type .llcrypt registered [ 1074.298888] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 15:33:47 (1788464027) [ 1076.038256] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1077.763326] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 15:33:51 (1788464031) [ 1078.432881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.454960] alg: No test for adler32 (adler32-zlib) [ 1079.459236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1079.526728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1079.560932] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1080.456861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.33 [ 1080.486761] LNet: 54433:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1080.507396] LNet: Removed LNI 192.168.203.33@tcp [ 1087.092916] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 15:34:00 (1788464040) [ 1087.387362] LNet: 55003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.405369] LNetError: 55003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.422740] LNet: Removed LNI 192.168.203.33@tcp [ 1088.026365] Key type .llcrypt unregistered [ 1088.029042] Key type ._llcrypt unregistered [ 1095.292853] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1095.492135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1098.274795] Key type ._llcrypt registered [ 1098.279647] Key type .llcrypt registered [ 1098.428473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1098.449348] alg: No test for adler32 (adler32-zlib) [ 1099.840072] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1106.789578] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 15:34:20 (1788464060) [ 1106.930640] LNet: 55907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.948141] LNetError: 55907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1106.978371] LNet: Removed LNI 10.1.2.3@tcp [ 1107.712552] Key type .llcrypt unregistered [ 1107.717294] Key type ._llcrypt unregistered [ 1114.843837] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1115.181782] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1115.189247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1115.202994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1118.113925] Key type ._llcrypt registered [ 1118.121692] Key type .llcrypt registered [ 1118.398929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.444814] alg: No test for adler32 (adler32-zlib) [ 1119.532111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1126.724525] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 15:34:40 (1788464080) [ 1128.474577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1130.621576] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 15:34:44 (1788464084) [ 1130.903725] LNet: 57096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.913527] LNetError: 57096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1130.935862] LNet: Removed LNI 10.1.2.3@tcp [ 1131.719764] Key type .llcrypt unregistered [ 1131.722200] Key type ._llcrypt unregistered [ 1138.140128] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1138.420309] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1138.429758] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1138.441735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1141.081624] Key type ._llcrypt registered [ 1141.083807] Key type .llcrypt registered [ 1141.254093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.271651] alg: No test for adler32 (adler32-zlib) [ 1142.269303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.363934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1150.891719] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 15:35:04 (1788464104) [ 1151.282553] LNet: 58323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.298439] LNetError: 58323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.315445] LNet: Removed LNI 10.1.2.3@tcp [ 1151.878631] Key type .llcrypt unregistered [ 1151.882613] Key type ._llcrypt unregistered [ 1159.096595] Key type ._llcrypt registered [ 1159.098911] Key type .llcrypt registered [ 1159.329514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1159.371308] alg: No test for adler32 (adler32-zlib) [ 1160.345984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1160.402378] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1168.923505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1169.630072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_interrupt -r 1 [ 1169.670580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_interrupt -r 1 [ 1169.720001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1169.737747] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1169.922452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.946591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.968641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.995852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.028575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1170.061331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.098332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.121218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.147769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.161627] LNet: 58761:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.186469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.231736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.270693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.302695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.324529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.360271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.395424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1170.855471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_dropped -r 1 [ 1170.916560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_dropped -r 1 [ 1170.955219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1170.970930] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1171.172728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.202556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.238500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.277658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.310901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1171.339275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.368278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.394865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.438594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.463654] LNet: 58872:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.487159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.521285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.563390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.590857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.632497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.663496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.681411] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1171.705911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.441916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_aborted -r 1 [ 1172.499267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_aborted -r 1 [ 1172.556403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1172.580945] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1172.586742] LNet: Skipped 1 previous similar message [ 1172.871850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1172.913413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1172.962404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.013864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.063687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1173.111646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.155095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.218806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.261310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.296503] LNet: 58984:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.316939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.373488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.432509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.491887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.541905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.573734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.611407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1174.307527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_no_route -r 1 [ 1174.377702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_no_route -r 1 [ 1174.448050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1174.671298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.692395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.718620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.739733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.768081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1174.797617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.834642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.868545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.915607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.946513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.981685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.025738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.074947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.134612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.183583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.238877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1175.739346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_timeout -r 1 [ 1175.797898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_timeout -r 1 [ 1175.838678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1175.863688] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1175.868573] LNet: Skipped 1 previous similar message [ 1176.104821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.134957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.160552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.185646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.208742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1176.235335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.270641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.322961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.350819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.371597] LNet: 59207:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.377490] LNet: 59207:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1176.395311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.443560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.474811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.500610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.572648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.630901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.683278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.312637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_error -r 1 [ 1177.346251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_error -r 1 [ 1177.390518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1177.648386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.682350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.737274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.765176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.817382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1177.850630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.893344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.932442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.998247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.064861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.124011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.177870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.232204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.278179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.324727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.370472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.648501] LNet: 59359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.655858] LNetError: 59359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.691046] LNet: Removed LNI 192.168.203.33@tcp [ 1180.101473] Key type .llcrypt unregistered [ 1180.106724] Key type ._llcrypt unregistered [ 1198.313433] Key type ._llcrypt registered [ 1198.317794] Key type .llcrypt registered [ 1199.589638] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 15:35:53 (1788464153) [ 1200.634518] Key type .llcrypt unregistered [ 1200.638773] Key type ._llcrypt unregistered [ 1207.077630] Key type ._llcrypt registered [ 1207.080710] Key type .llcrypt registered [ 1207.253543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1207.268447] alg: No test for adler32 (adler32-zlib) [ 1208.203719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1208.247394] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1216.882148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1227.732378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.758815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1227.785149] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1229.199201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_interrupt -r 1 [ 1229.249759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 1 [ 1229.294914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_interrupt -r 1 [ 1229.342789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 1 [ 1229.390804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_interrupt -r 1 [ 1229.439979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 1 [ 1229.492989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_interrupt -r 1 [ 1229.539070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 1 [ 1229.575448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1229.590680] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1229.597081] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1229.796747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.826875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1229.877814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.925859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.957431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1229.994867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.026346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.058746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.095290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.128227] LNet: 60826:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.164573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.206425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.237911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.270306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.326611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.369179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.404196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.823269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_dropped -r 1 [ 1230.855351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 1 [ 1230.887654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_dropped -r 1 [ 1230.907642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 1 [ 1230.925933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_dropped -r 1 [ 1230.953077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 1 [ 1230.977510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_dropped -r 1 [ 1230.998945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 1 [ 1231.023230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1231.035657] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1231.038578] LNet: Skipped 2 previous similar messages [ 1231.249072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.270559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1231.287957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.307656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.332085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1231.357693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.414081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.454859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.481437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.498566] LNet: 60949:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.520526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.547882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.568068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1231.582269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1231.602068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1231.620648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.642398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1232.031637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_aborted -r 1 [ 1232.066095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 1 [ 1232.098199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_aborted -r 1 [ 1232.135652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 1 [ 1232.171785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_aborted -r 1 [ 1232.198121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 1 [ 1232.227279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_aborted -r 1 [ 1232.269514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 1 [ 1232.298148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1232.320879] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1232.328994] LNet: Skipped 2 previous similar messages [ 1232.510442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.534772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1232.565549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.596362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.628450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1232.666086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.710602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1232.749948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1232.781956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1232.794551] LNet: 61072:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1232.808876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1232.844548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1232.872499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.907323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.938485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.980330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1233.027863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1233.553652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_no_route -r 1 [ 1233.594112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 1 [ 1233.616990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_no_route -r 1 [ 1233.640937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 1 [ 1233.660293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_no_route -r 1 [ 1233.707570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 1 [ 1233.753470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_no_route -r 1 [ 1233.804601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 1 [ 1233.850512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1234.047972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.073793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1234.097953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.130273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.165531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1234.213844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.247621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.288330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.355460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.378939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.408860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.446765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.491829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.525193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.561204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.587195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1235.003412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_timeout -r 1 [ 1235.054387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 1 [ 1235.107197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_timeout -r 1 [ 1235.148477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 1 [ 1235.188693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_timeout -r 1 [ 1235.221900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 1 [ 1235.267610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_timeout -r 1 [ 1235.318144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 1 [ 1235.385873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1235.404207] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1235.408820] LNet: Skipped 5 previous similar messages [ 1235.638859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.673794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.708462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.743148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.779933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1235.809757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.846698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.883232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.920336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.947313] LNet: 61319:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.955681] LNet: 61319:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1235.972734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.001279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.026128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.066163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.100270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.133721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.179305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.883236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_error -r 1 [ 1236.925835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_error -r 1 [ 1236.971643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_error -r 1 [ 1236.997536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_error -r 1 [ 1237.030469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_error -r 1 [ 1237.057840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_error -r 1 [ 1237.085433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_error -r 1 [ 1237.124522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_error -r 1 [ 1237.166533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1237.476711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.509767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.561302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.593391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.640995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1237.673227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.704440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.738936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.802544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.854959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.893535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.931849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.960971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.985438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.036763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.073388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.104540] LNet: 61518:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1244.117348] LNetError: 61518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1244.122956] LNet: Removed LNI 192.168.203.33@tcp [ 1244.148437] LNet: Removed LNI 192.168.203.33@tcp1 [ 1244.751867] Key type .llcrypt unregistered [ 1244.754320] Key type ._llcrypt unregistered [ 1263.827864] Key type ._llcrypt registered [ 1263.829649] Key type .llcrypt registered [ 1264.999341] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 15:36:58 (1788464218) [ 1266.510684] Key type .llcrypt unregistered [ 1266.518198] Key type ._llcrypt unregistered [ 1274.561332] Key type ._llcrypt registered [ 1274.565574] Key type .llcrypt registered [ 1274.823457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1274.845333] alg: No test for adler32 (adler32-zlib) [ 1275.726040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1275.759891] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1284.834163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1285.791481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_dropped -r 1 [ 1285.848547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_dropped -r 1 [ 1285.896303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1285.912872] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1286.243880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1286.280567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1286.315412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1286.350887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1286.397703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1286.437780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1286.473791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1286.516814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1286.550851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1286.578677] LNet: 62767:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1286.602092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1286.657806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1286.736357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1286.789686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1286.853048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1286.927365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1286.989695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1287.533415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_error -r 1 [ 1287.561465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_error -r 1 [ 1287.606688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1287.627564] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1287.954584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.984925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.031468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.072540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.111552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1288.157672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.201203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.258913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.320333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.345755] LNet: 62878:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.365960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.406344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.477093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.524981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.587329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.634917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.684770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1289.300881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_timeout -r 1 [ 1289.355744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_timeout -r 1 [ 1289.413820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1289.434395] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1289.701181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.729843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.751460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.785526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.824777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1289.876175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.916319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.946076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.015494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.048702] LNet: 62989:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.078697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.113640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.142889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.175829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.213445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.246659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.297308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1290.604365] LNet: 63030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.611750] LNetError: 63030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.635667] LNet: Removed LNI 192.168.203.33@tcp [ 1291.172558] Key type .llcrypt unregistered [ 1291.175656] Key type ._llcrypt unregistered [ 1310.617864] Key type ._llcrypt registered [ 1310.619862] Key type .llcrypt registered [ 1312.069767] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 15:37:45 (1788464265) [ 1313.136714] Key type .llcrypt unregistered [ 1313.143510] Key type ._llcrypt unregistered [ 1320.231418] Key type ._llcrypt registered [ 1320.233964] Key type .llcrypt registered [ 1320.563833] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1320.582812] alg: No test for adler32 (adler32-zlib) [ 1321.523362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1321.549253] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1329.781577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1340.316383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.355684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1340.374319] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1341.980780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_dropped -r 1 [ 1342.032267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 1 [ 1342.069977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_dropped -r 1 [ 1342.105218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 1 [ 1342.140891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_dropped -r 1 [ 1342.233976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 1 [ 1342.281487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_dropped -r 1 [ 1342.328840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 1 [ 1342.376825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1342.418402] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1342.426306] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 1342.721334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.745312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.770888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.799155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.830473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1342.864406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.896590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.940825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.977421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.996384] LNet: 64497:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1343.011344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1343.036574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1343.070388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.100388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1343.137520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.168370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.206157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1343.232671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1343.908699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_error -r 1 [ 1343.967395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_error -r 1 [ 1344.024684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_error -r 1 [ 1344.068571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_error -r 1 [ 1344.121948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_error -r 1 [ 1344.165671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_error -r 1 [ 1344.212319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_error -r 1 [ 1344.261937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_error -r 1 [ 1344.329384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1344.355457] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1344.361389] LNet: Skipped 2 previous similar messages [ 1344.600769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1344.635521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1344.661500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1344.718309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1344.760667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1344.802221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1344.837574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1344.864254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1344.893902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1344.928444] LNet: 64622:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1344.957924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1344.999163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1345.044835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.083277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1345.137764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.217367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1345.262829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1345.306729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1345.942233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_timeout -r 1 [ 1345.997160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_timeout -r 1 [ 1346.042457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_timeout -r 1 [ 1346.072793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_timeout -r 1 [ 1346.103036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_timeout -r 1 [ 1346.143389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_timeout -r 1 [ 1346.173683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_timeout -r 1 [ 1346.233142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_timeout -r 1 [ 1346.275455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.133@tcp [ 1346.301902] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1346.565812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1346.595156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1346.619305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1346.646953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1346.685384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1346.727936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1346.763137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1346.810982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1346.854963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1346.886701] LNet: 64748:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1346.903639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1346.929639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1346.962610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.000222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.032501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.070922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1347.116336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1347.148361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1352.594451] LNet: 64826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.600338] LNetError: 64826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.609445] LNet: Removed LNI 192.168.203.33@tcp [ 1352.636677] LNet: Removed LNI 192.168.203.33@tcp1 [ 1353.024156] Key type .llcrypt unregistered [ 1353.028212] Key type ._llcrypt unregistered [ 1372.296256] Key type ._llcrypt registered [ 1372.301585] Key type .llcrypt registered [ 1373.736590] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 15:38:47 (1788464327) [ 1375.482343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1377.241811] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 15:38:51 (1788464331) [ 1378.252277] Key type .llcrypt unregistered [ 1378.256648] Key type ._llcrypt unregistered [ 1385.687861] Key type ._llcrypt registered [ 1385.689500] Key type .llcrypt registered [ 1385.989411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1386.014561] alg: No test for adler32 (adler32-zlib) [ 1386.961605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1386.996440] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1396.427492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1397.551816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e network_timeout -r 1 [ 1397.622594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e network_timeout -r 1 [ 1397.725259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1397.811086] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1398.145340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1398.175349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1398.222900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1398.262621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1398.305118] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1398.308656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1398.356668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1398.394668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1398.439765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1398.500041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1398.536719] LNet: 66369:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1398.559828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1398.615811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1398.664445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1398.712948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1398.764664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1398.814873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.247641] LNet: 66408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1399.269699] LNetError: 66408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1399.302822] LNet: Removed LNI 192.168.203.33@tcp [ 1399.662480] Key type .llcrypt unregistered [ 1399.667350] Key type ._llcrypt unregistered [ 1420.976175] Key type ._llcrypt registered [ 1420.979469] Key type .llcrypt registered [ 1421.189176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.204718] alg: No test for adler32 (adler32-zlib) [ 1422.189875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1422.235339] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1431.017659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1440.425155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.483930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1440.517781] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1441.988918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e network_timeout -r 1 [ 1442.025497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e network_timeout -r 1 [ 1442.057742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e network_timeout -r 1 [ 1442.089747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e network_timeout -r 1 [ 1442.173874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e network_timeout -r 1 [ 1442.241759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e network_timeout -r 1 [ 1442.276815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e network_timeout -r 1 [ 1442.312755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e network_timeout -r 1 [ 1442.371659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1442.405337] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1442.721049] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1442.723639] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 1442.818469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1442.842210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1442.888611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1442.926886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1442.959774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1442.997539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1443.027638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1443.071802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1443.129043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1443.159461] LNet: 67286:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1443.190602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1443.236082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1443.325858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1443.365933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1443.395849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1443.418415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1448.851333] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1449.317773] LNet: 67361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1449.327410] LNetError: 67361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1449.346683] LNet: Removed LNI 192.168.203.33@tcp [ 1449.356925] LNet: Removed LNI 192.168.203.33@tcp1 [ 1449.944543] Key type .llcrypt unregistered [ 1449.946439] Key type ._llcrypt unregistered [ 1468.611782] Key type ._llcrypt registered [ 1468.615711] Key type .llcrypt registered [ 1470.029528] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 15:40:23 (1788464423) [ 1470.302560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1470.320839] alg: No test for adler32 (adler32-zlib) [ 1471.193630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.236776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.270170] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1471.992829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1472.014871] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1472.708409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1472.767111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1472.800175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1472.842331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1472.893908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1472.929804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1472.954463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.33@tcp [ 1472.966373] LNetError: 68458:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 1473.249192] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1473.252392] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1475.296383] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1476.443338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1479.392447] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1479.986634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1480.036722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1480.108189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.135414] LNet: 68565:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.143901] LNetError: 68565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1480.149634] LNet: Removed LNI 192.168.203.33@tcp [ 1480.164184] LNet: Removed LNI 192.168.203.33@tcp1 [ 1480.207988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.264137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.295321] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1481.168888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.197923] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1481.861728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1481.897319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.965284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1482.017143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1482.061810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1482.108630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1482.136269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1482.158387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.33@tcp [ 1482.180194] LNetError: 68838:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 1482.278903] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1484.748837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1488.416572] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1488.422208] LNet: Skipped 1 previous similar message [ 1493.186942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1493.226270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1493.266359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1501.262922] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 15:40:55 (1788464455) [ 1501.597438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.623489] LNet: 69546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.631653] LNetError: 69546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.639555] LNetError: 69546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1501.647544] LNet: Removed LNI 192.168.203.33@tcp [ 1501.667486] LNet: Removed LNI 192.168.203.33@tcp1 [ 1501.722636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.793443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.820749] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1502.711737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.737596] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1503.539146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1503.685976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1503.736561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1503.783428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1503.834763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1503.876460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1503.912092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.33@tcp --health 0 [ 1503.998862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.800324] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.33@tcp [ 1504.803527] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1504.803535] LNet: Skipped 1 previous similar message [ 1518.240904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1518.434239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1518.463145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1518.491231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.33@tcp --health 500 [ 1518.673447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1525.221672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1525.253992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1525.268605] LNet: 70030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1525.273052] LNetError: 70030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1525.276318] LNetError: 70030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1525.282442] LNet: Removed LNI 192.168.203.33@tcp [ 1525.291516] LNet: Removed LNI 192.168.203.33@tcp1 [ 1525.309376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1525.344718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1525.366787] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1525.988785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1526.018994] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1526.902782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1526.943995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1527.042326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1527.065830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1527.095830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1527.124132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1527.146353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1527.181363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.33@tcp --health 0 [ 1528.416945] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1528.427432] LNet: Skipped 3 previous similar messages [ 1529.647245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1538.949434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1538.985683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1539.027777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1547.455804] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 15:41:41 (1788464501) [ 1548.256607] LNet: 71077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.266067] LNetError: 71077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.271541] LNetError: 71077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1548.276876] LNet: Removed LNI 192.168.203.33@tcp [ 1548.291536] LNet: Removed LNI 192.168.203.33@tcp1 [ 1549.076835] Key type .llcrypt unregistered [ 1549.080728] Key type ._llcrypt unregistered [ 1555.999609] Key type ._llcrypt registered [ 1556.002152] Key type .llcrypt registered [ 1556.221583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1556.242924] alg: No test for adler32 (adler32-zlib) [ 1557.198110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1557.231444] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1567.171324] Lustre: DEBUG MARKER: Initial discovery [ 1567.231878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.133@tcp [ 1569.874723] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1569.993964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1570.052161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.133@tcp [ 1570.073995] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1570.090618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1570.123596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.133@tcp -r 1 -m GET -l 3 [ 1570.163712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.133@tcp -r 1 -m GET -e local_error [ 1570.213285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.133@tcp -r 1 -m PUT -l 6 [ 1570.530307] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1571.415598] Lustre: DEBUG MARKER: Force oleg333-client.virtnet to discover 192.168.203.133@tcp (in background) [ 1571.674366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.133@tcp [ 1572.688725] Lustre: DEBUG MARKER: Force oleg333-server to discover 192.168.203.33@tcp [ 1574.756155] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1575.971638] Lustre: DEBUG MARKER: Wait for 71837 [ 1577.429315] Lustre: DEBUG MARKER: Finished wait on 71837 [ 1577.698111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1577.758553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1577.760122] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1578.313562] LNet: 72286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1578.329633] LNetError: 72286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1578.363675] LNet: Removed LNI 192.168.203.33@tcp [ 1578.798952] Key type .llcrypt unregistered [ 1578.802599] Key type ._llcrypt unregistered [ 1595.235684] Key type ._llcrypt registered [ 1595.238088] Key type .llcrypt registered [ 1596.649388] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 15:42:30 (1788464550) [ 1597.585607] Key type .llcrypt unregistered [ 1597.588043] Key type ._llcrypt unregistered [ 1605.077818] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1605.168746] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1605.181426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1605.203535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1606.840693] Key type ._llcrypt registered [ 1606.844259] Key type .llcrypt registered [ 1607.085436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1607.121352] alg: No test for adler32 (adler32-zlib) [ 1608.026824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.083675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1608.107659] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1608.697250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1608.714418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1617.911556] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 15:42:51 (1788464571) [ 1618.008554] LNet: 74346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1618.019592] LNetError: 74346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1618.024758] LNet: Removed LNI 10.1.2.3@tcp [ 1618.037563] LNet: Removed LNI 192.168.203.33@tcp [ 1618.770557] Key type .llcrypt unregistered [ 1618.774213] Key type ._llcrypt unregistered [ 1625.889866] Key type ._llcrypt registered [ 1625.896193] Key type .llcrypt registered [ 1626.148535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1626.177503] alg: No test for adler32 (adler32-zlib) [ 1627.199283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.269766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.320395] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1628.292417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1628.330991] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1629.469407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.33@tcp [ 1630.560224] LNetError: Unexpected error -22 connecting to 192.168.203.33@tcp at host 192.168.203.33:988 [ 1635.925427] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 15:43:09 (1788464589) [ 1636.307320] LNet: 75482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.331379] LNetError: 75482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1636.347556] LNet: Removed LNI 10.1.2.3@tcp [ 1636.369625] LNet: Removed LNI 192.168.203.33@tcp [ 1637.468513] Key type .llcrypt unregistered [ 1637.470135] Key type ._llcrypt unregistered [ 1644.080230] Key type ._llcrypt registered [ 1644.083140] Key type .llcrypt registered [ 1644.256984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1644.280501] alg: No test for adler32 (adler32-zlib) [ 1645.266914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.301511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1645.321950] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1646.067812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1646.103839] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 1646.834736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.33@tcp1 --nid 192.168.203.33@tcp2 [ 1651.286502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp1 192.168.203.33@tcp2 [ 1651.304647] LNetError: 76314:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.33@tcp1 to 192.168.203.33@tcp2 - Route table may be misconfigured [ 1651.320830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp2 192.168.203.33@tcp1 [ 1658.332266] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 15:43:32 (1788464612) [ 1658.426116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.462453] LNet: 76883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.472321] LNetError: 76883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.488138] LNet: Removed LNI 192.168.203.33@tcp1 [ 1659.490474] LNetError: 76883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.524794] LNet: Removed LNI 192.168.203.33@tcp2 [ 1659.565509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.621940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1659.654374] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1660.509770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1660.522656] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1661.418360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 1661.467807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.33@tcp [ 1661.547544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp1 192.168.203.33@tcp1 [ 1661.624293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.33@tcp -d 192.168.203.33@tcp -e network_timeout [ 1661.670586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -e network_timeout [ 1661.709221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -e network_timeout [ 1661.742072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -e network_timeout [ 1661.772273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1661.803436] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1661.835908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1661.971493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1662.098528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1662.689317] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1668.680413] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 15:43:42 (1788464622) [ 1668.867110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.903380] LNet: 77751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.918881] LNetError: 77751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.925748] LNet: Removed LNI 192.168.203.33@tcp [ 1669.993146] LNet: Removed LNI 192.168.203.33@tcp1 [ 1670.034822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.147652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1670.422801] LNet: 77789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.816489] Key type .llcrypt unregistered [ 1670.824394] Key type ._llcrypt unregistered [ 1682.966608] Key type ._llcrypt registered [ 1682.970297] Key type .llcrypt registered [ 1684.532842] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 15:43:57 (1788464637) [ 1685.070905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1685.099472] alg: No test for adler32 (adler32-zlib) [ 1686.115608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.334489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1688.990959] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 15:44:02 (1788464642) [ 1689.128850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.192976] LNet: 78878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.234423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.343975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.398372] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1690.255977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1690.280969] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1691.414850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1691.444807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp1 [ 1691.514626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp1 [ 1698.540674] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 15:44:12 (1788464652) [ 1700.979910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg333-server [ 1704.029709] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 15:44:16 (1788464656) [ 1705.966915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg333-server [ 1707.628821] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 15:44:21 (1788464661) [ 1710.216089] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg333-server [ 1712.238786] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 15:44:26 (1788464666) [ 1715.254088] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg333-server [ 1718.250432] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 15:44:31 (1788464671) [ 1722.151127] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg333-server [ 1724.161919] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 15:44:37 (1788464677) [ 1727.350598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg333-server [ 1729.719296] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 15:44:42 (1788464682) [ 1732.393593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg333-server [ 1734.749639] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 15:44:48 (1788464688) [ 1738.123208] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg333-server [ 1740.438855] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 15:44:53 (1788464693) [ 1743.232961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg333-server [ 1745.924490] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 15:44:58 (1788464698) [ 1746.186296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.245087] LNet: 82760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.258602] LNetError: 82760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.286983] LNet: Removed LNI 192.168.203.33@tcp [ 1747.318626] LNet: Removed LNI 192.168.203.33@tcp1 [ 1747.348757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.395812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.431671] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1747.905955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1747.973785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1748.205218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.249047] LNet: 82914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.262844] LNetError: 82914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.275715] LNetError: 82914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1748.334080] LNet: Removed LNI 192.168.203.33@tcp [ 1748.399473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.540519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1748.586972] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1749.562387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1749.714584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.765090] LNetError: 83059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.781440] LNet: Removed LNI 192.168.203.33@tcp [ 1749.844838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.984720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.009313] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1751.034520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1751.097401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1751.305202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.348657] LNet: 83207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1751.355256] LNet: 83207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1752.429461] LNet: Removed LNI 192.168.203.33@tcp [ 1752.472592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.583830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1752.656614] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1753.487590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1753.762660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.813094] LNetError: 83352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.825314] LNetError: 83352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1753.849219] LNet: Removed LNI 192.168.203.33@tcp [ 1753.885446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.992659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.029461] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1754.868653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1754.972879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1755.188034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.264373] LNet: Removed LNI 192.168.203.33@tcp [ 1756.343622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.375710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1756.404651] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1757.160532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1757.346150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.399815] LNet: 83645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.414839] LNet: 83645:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1758.438958] LNet: Removed LNI 192.168.203.33@tcp [ 1758.474657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.515860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.532114] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1759.231597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1759.338928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1759.606441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1759.667047] LNetError: 83792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1759.679800] LNetError: 83792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1759.701863] LNet: Removed LNI 192.168.203.33@tcp [ 1759.746531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1759.815859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1759.851655] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1760.800292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1761.143438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.187034] LNet: Removed LNI 192.168.203.33@tcp [ 1761.240332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.331821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1761.382853] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1762.151645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1762.231447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1762.458423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1763.499809] LNet: Removed LNI 192.168.203.33@tcp [ 1763.528895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.602678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1763.650172] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1764.523520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1764.829928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.928457] LNet: Removed LNI 192.168.203.33@tcp [ 1765.969561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.052948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.081452] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1766.832808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1766.903488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1767.139455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.175855] LNet: 84378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.181088] LNet: 84378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1768.233151] LNet: Removed LNI 192.168.203.33@tcp [ 1768.279174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.347691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1768.390962] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1769.067146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 1769.363543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.393849] LNetError: 84523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.399227] LNetError: 84523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1769.414319] LNet: Removed LNI 192.168.203.33@tcp [ 1769.432684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.494096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.532601] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1770.273158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1770.332603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1770.362595] LNet: Removed LNI 192.168.203.33@tcp [ 1770.391265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1770.446321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1770.468456] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1771.147979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1771.215399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.248672] LNet: Removed LNI 192.168.203.33@tcp [ 1771.277137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.343955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1771.437951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.480355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.541222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1771.598789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.635238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.703546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1771.788298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.831708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.871162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1771.892088] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1772.535349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.561515] LNet: Removed LNI 192.168.203.33@tcp [ 1772.583532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.669375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1779.088164] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 15:45:33 (1788464733) [ 1779.334430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.372437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.446789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1779.479653] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 1780.703569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.753774] LNet: Removed LNI 192.168.203.33@tcp231 [ 1780.776644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.854557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1780.874929] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 1780.879527] LNet: 85690:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.226789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1781.258114] LNet: Removed LNI 192.168.203.33@tcp231 [ 1781.278739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1781.309890] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 1782.417554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.452674] LNet: Removed LNI 192.168.203.33@tcp231 [ 1782.469877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.536125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1782.559667] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/0] [ 1782.564141] LNet: 85843:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1782.861698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1782.901636] LNet: Removed LNI 192.168.203.33@tcp231 [ 1782.918879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1782.939717] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/0] [ 1783.886754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.914777] LNet: 85986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.922966] LNet: 85986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1783.946886] LNet: Removed LNI 192.168.203.33@tcp231 [ 1783.971785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.033229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1784.058933] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/60] [ 1784.066549] LNet: 85994:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1784.448383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1784.483024] LNet: Removed LNI 192.168.203.33@tcp231 [ 1784.505339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1784.533768] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/60] [ 1785.497243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.525111] LNetError: 86137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1785.529339] LNetError: 86137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1785.546136] LNet: Removed LNI 192.168.203.33@tcp231 [ 1785.579247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.670331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1785.692721] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 1786.103626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1786.130905] LNet: Removed LNI 192.168.203.33@tcp231 [ 1786.144374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1786.175469] LNet: Added LNI 192.168.203.33@tcp231 [8/256/0/180] [ 1793.089242] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 15:45:46 (1788464746) [ 1793.358560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.410207] LNet: Removed LNI 192.168.203.33@tcp231 [ 1793.452562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.521119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1793.549813] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 1794.238650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1794.272862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1794.299783] LNet: Removed LNI 192.168.203.33@tcp232 [ 1794.320127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1794.368449] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 1795.029394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1795.075608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1795.118187] LNet: Removed LNI 192.168.203.33@tcp232 [ 1795.137614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1795.179607] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 1795.910260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1795.945849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1795.977617] LNet: Removed LNI 192.168.203.33@tcp232 [ 1796.015946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1796.052715] LNet: Added LNI 192.168.203.33@tcp232 [8/256/0/180] [ 1796.885614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1796.948609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1796.983501] LNet: Removed LNI 192.168.203.33@tcp232 [ 1797.012060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1802.585320] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 15:45:56 (1788464756) [ 1803.550193] Key type .llcrypt unregistered [ 1803.552815] Key type ._llcrypt unregistered [ 1809.125384] Key type ._llcrypt registered [ 1809.127947] Key type .llcrypt registered [ 1809.286780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1809.298336] alg: No test for adler32 (adler32-zlib) [ 1810.210483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1810.255640] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1817.387395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1825.899700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.988691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1826.047427] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1827.643651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_interrupt -r 2 [ 1827.702783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 2 [ 1827.760546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_interrupt -r 2 [ 1827.804822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 2 [ 1827.844809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_interrupt -r 2 [ 1827.894295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_interrupt -r 2 [ 1827.945175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_interrupt -r 2 [ 1828.008034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_interrupt -r 2 [ 1828.072732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 1828.132198] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1828.147568] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1828.470495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1828.532767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1828.564575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1828.625845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1828.686758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1828.728244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1828.769381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1828.832739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1828.874665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1828.898870] LNet: 88646:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1828.934269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1828.991714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1829.033854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1829.094129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1829.163930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1829.226817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1830.018515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_dropped -r 2 [ 1830.083851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 2 [ 1830.130138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_dropped -r 2 [ 1830.179096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 2 [ 1830.271989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_dropped -r 2 [ 1830.396692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_dropped -r 2 [ 1830.451890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_dropped -r 2 [ 1830.509440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_dropped -r 2 [ 1830.597749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 1830.640715] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1830.969762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1831.003994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1831.032177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1831.070069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1831.114808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1831.156862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1831.189335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1831.241501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1831.288404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1831.309531] LNet: 88768:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1831.342524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1831.388075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1831.419867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1831.444279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1831.473849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1831.502629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1832.066642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_aborted -r 2 [ 1832.119974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 2 [ 1832.148674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_aborted -r 2 [ 1832.200716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 2 [ 1832.259936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_aborted -r 2 [ 1832.316609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_aborted -r 2 [ 1832.373304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_aborted -r 2 [ 1832.403698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_aborted -r 2 [ 1832.432802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 1832.450362] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1832.707750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1832.733677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1832.764459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1832.797729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1832.828887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1832.857598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1832.875841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1832.907667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1832.936986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1832.951064] LNet: 88891:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1832.973493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1832.999907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1833.052609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1833.089024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1833.120110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1833.167195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1833.569578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_no_route -r 2 [ 1833.593575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 2 [ 1833.618667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_no_route -r 2 [ 1833.643345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 2 [ 1833.667756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_no_route -r 2 [ 1833.691966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_no_route -r 2 [ 1833.719701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_no_route -r 2 [ 1833.747616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_no_route -r 2 [ 1833.776383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 1833.996695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1834.028732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1834.056764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1834.078616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1834.106386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1834.128348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1834.153964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1834.178167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1834.202238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1834.225855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1834.250678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1834.275608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1834.315129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1834.347282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1834.376800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1834.848845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e local_timeout -r 2 [ 1834.885969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 2 [ 1834.936790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e local_timeout -r 2 [ 1834.979564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 2 [ 1835.020685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e local_timeout -r 2 [ 1835.055541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e local_timeout -r 2 [ 1835.083270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e local_timeout -r 2 [ 1835.117744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e local_timeout -r 2 [ 1835.152340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 1835.169672] LNet: There was an unexpected network error while writing to 192.168.203.133: rc = -22 [ 1835.173828] LNet: Skipped 1 previous similar message [ 1835.442536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1835.492771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1835.546049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1835.602504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1835.680774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1835.750822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1835.799655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1835.863964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1835.922328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1835.962736] LNet: 89135:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1835.977905] LNet: 89135:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1836.029258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1836.071326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1836.093679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1836.120958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1836.151902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1836.179974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1836.868225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp -m GET -e remote_dropped -r 2 [ 1836.924627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 2 [ 1836.960367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp -m GET -e remote_dropped -r 2 [ 1836.991580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 2 [ 1837.026544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp -m GET -e remote_dropped -r 2 [ 1837.064618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.133@tcp1 -m GET -e remote_dropped -r 2 [ 1837.101884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp -m GET -e remote_dropped -r 2 [ 1837.130544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.33@tcp1 -d 192.168.203.33@tcp1 -m GET -e remote_dropped -r 2 [ 1837.177799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.33@tcp 192.168.203.133@tcp [ 1837.205936] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 1837.476621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1837.512461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1837.599069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1837.662565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1837.728932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1837.792217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1837.833904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1837.873052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1837.904604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1837.927780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1837.971345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1837.996940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1838.039927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1838.088297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1838.134521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1844.179203] LNet: 89332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.188884] LNetError: 89332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.197566] LNet: Removed LNI 192.168.203.33@tcp [ 1844.213620] LNet: Removed LNI 192.168.203.33@tcp1 [ 1844.466607] Key type .llcrypt unregistered [ 1844.468054] Key type ._llcrypt unregistered [ 1860.754081] Key type ._llcrypt registered [ 1860.758766] Key type .llcrypt registered [ 1861.896880] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 15:46:55 (1788464815) [ 1862.579992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1862.595561] alg: No test for adler32 (adler32-zlib) [ 1863.496534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.541750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1863.572769] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1864.445430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1864.474673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.33@tcp [ 1864.503684] LNetError: 90309:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 1864.545529] LNet: 1 local NIs in recovery (showing 1): 192.168.203.33@tcp [ 1864.626215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.33@tcp [ 1864.819500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1871.473379] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 15:47:05 (1788464825) [ 1871.742830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1871.777682] LNet: 90899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1871.784904] LNetError: 90899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.807439] LNet: Removed LNI 192.168.203.33@tcp [ 1871.830606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1871.925707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1871.942074] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1872.677049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1872.696618] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 1873.782676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.33@tcp [ 1873.801565] LNetError: 91165:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.33@tcp to recovery [ 1873.957308] LNet: There was an unexpected network error while writing to 192.168.203.33: rc = -22 [ 1875.040291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.33@tcp [ 1883.063591] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 15:47:16 (1788464836) [ 1885.547639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg333-server [ 1887.575853] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 15:47:21 (1788464841) [ 1890.078800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg333-server [ 1891.480561] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 15:47:25 (1788464845) [ 1893.547979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg333-server [ 1895.042889] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 15:47:29 (1788464849) [ 1895.322612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.346480] LNet: 92788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.359718] LNetError: 92788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.367624] LNet: Removed LNI 192.168.203.33@tcp [ 1895.382614] LNet: Removed LNI 192.168.203.33@tcp1 [ 1895.409939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.499482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1895.534796] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1896.287263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1896.365378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1896.416396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1896.575599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1896.640548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1896.711944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1904.760597] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 15:47:37 (1788464857) [ 1904.982149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.008493] LNet: 93517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.016911] LNetError: 93517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.026156] LNetError: 93517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1905.058526] LNet: Removed LNI 192.168.203.33@tcp [ 1905.087346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.153960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.170305] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1915.657953] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 15:47:48 (1788464868) [ 1915.819733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.862578] LNet: 94217:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.876574] LNetError: 94217:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.933223] LNet: Removed LNI 192.168.203.33@tcp [ 1915.979882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.122827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.187168] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1925.324199] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 15:47:58 (1788464878) [ 1927.191346] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1928.881587] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 15:48:02 (1788464882) [ 1929.696684] LNet: 95258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1929.707584] LNetError: 95258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.723716] LNet: Removed LNI 192.168.203.33@tcp [ 1930.443480] Key type .llcrypt unregistered [ 1930.445508] Key type ._llcrypt unregistered [ 1936.997925] Key type ._llcrypt registered [ 1937.003728] Key type .llcrypt registered [ 1937.242122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1937.251967] alg: No test for adler32 (adler32-zlib) [ 1938.171905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1938.227136] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1946.444633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 1951.153122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.133@tcp [ 1951.462381] LNet: 95674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.476765] LNetError: 95674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.490523] LNet: Removed LNI 192.168.203.33@tcp [ 1951.959291] Key type .llcrypt unregistered [ 1951.962160] Key type ._llcrypt unregistered [ 1962.997568] Key type ._llcrypt registered [ 1963.000210] Key type .llcrypt registered [ 1964.291073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1966.024618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1967.887207] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 15:48:41 (1788464921) [ 1969.324250] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1971.135547] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 15:48:45 (1788464925) [ 1973.250218] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg333-server [ 1974.709514] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 15:48:48 (1788464928) [ 1977.222302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg333-server [ 1978.723255] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 15:48:52 (1788464932) [ 1979.948049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1980.051660] alg: No test for adler32 (adler32-zlib) [ 1981.084896] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 1981.365226] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 1983.008605] Key type lgssc registered [ 1984.190869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2009.109398] Key type lgssc unregistered [ 2009.433696] LNet: 98845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.442354] LNetError: 98845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.460391] LNet: Removed LNI 192.168.203.33@tcp [ 2010.178522] Key type .llcrypt unregistered [ 2010.181767] Key type ._llcrypt unregistered [ 2025.550945] Key type ._llcrypt registered [ 2025.553630] Key type .llcrypt registered [ 2027.035789] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 15:49:40 (1788464980) [ 2027.434390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2027.448881] alg: No test for adler32 (adler32-zlib) [ 2028.362247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2028.410557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.463979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.517776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2028.565867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2035.334590] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 15:49:49 (1788464989) [ 2035.471937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2035.522924] LNet: 100251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.558793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2035.656694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2035.704927] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2036.740480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.758927] LNet: 100401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.765094] LNetError: 100401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2036.804325] LNet: Removed LNI 192.168.203.33@tcp [ 2036.818103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.893529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2036.918453] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2037.368405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.401571] LNetError: 100429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.419925] LNet: Removed LNI 192.168.203.33@tcp [ 2037.443912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.491418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2037.516097] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2043.744950] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 15:49:57 (1788464997) [ 2044.016319] LNet: 101021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.021107] LNet: 101021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2044.027640] LNetError: 101021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.049202] LNet: Removed LNI 192.168.203.33@tcp [ 2044.718807] Key type .llcrypt unregistered [ 2044.722874] Key type ._llcrypt unregistered [ 2054.275346] Key type ._llcrypt registered [ 2054.277807] Key type .llcrypt registered [ 2054.882567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2054.920850] alg: No test for adler32 (adler32-zlib) [ 2055.875168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.954164] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2055.957895] LNet: *** cfs_fail_loc=e003, val=100*** [ 2056.886498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2056.911237] LNet: *** cfs_fail_loc=e003, val=100*** [ 2056.913309] LNet: Skipped 2 previous similar messages [ 2056.917134] Ping reply received with mapping: [ 2056.919473] found metadata reply NI[192.168.203.33@tcp] [ 2056.923518] 21 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2056.926404] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2056.928750] 5a 5a 5a 5a 5a 5a 5a 5a [ 2057.094829] LNet: 101416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2057.106479] LNetError: 101416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.124244] LNet: Removed LNI 192.168.203.33@tcp [ 2057.868402] Key type .llcrypt unregistered [ 2057.871248] Key type ._llcrypt unregistered [ 2071.520525] Key type ._llcrypt registered [ 2071.525345] Key type .llcrypt registered [ 2072.954452] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 15:50:26 (1788465026) [ 2073.429246] Key type .llcrypt unregistered [ 2073.433940] Key type ._llcrypt unregistered [ 2079.511362] Key type ._llcrypt registered [ 2079.513843] Key type .llcrypt registered [ 2079.677354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2079.687923] alg: No test for adler32 (adler32-zlib) [ 2080.759858] LNetError: 102452:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2081.126452] Key type .llcrypt unregistered [ 2081.128951] Key type ._llcrypt unregistered [ 2086.542820] Key type ._llcrypt registered [ 2086.547730] Key type .llcrypt registered [ 2087.730567] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 15:50:41 (1788465041) [ 2088.328393] Key type .llcrypt unregistered [ 2088.330235] Key type ._llcrypt unregistered [ 2094.348770] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2094.352645] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2096.830379] Key type ._llcrypt registered [ 2096.834590] Key type .llcrypt registered [ 2096.998216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2097.006582] alg: No test for adler32 (adler32-zlib) [ 2097.867680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.913194] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2105.966285] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 15:51:00 (1788465060) [ 2106.102245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.129969] LNet: 104220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.140317] LNetError: 104220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2106.173482] LNet: Removed LNI 10.1.2.3@tcp [ 2106.199583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.896842] Key type .llcrypt unregistered [ 2106.901281] Key type ._llcrypt unregistered [ 2114.154171] Key type ._llcrypt registered [ 2114.157786] Key type .llcrypt registered [ 2114.458952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.503722] alg: No test for adler32 (adler32-zlib) [ 2115.671344] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2115.827631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2116.154416] LNet: 104511:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.163235] LNetError: 104511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.178633] LNet: Removed LNI 192.168.203.33@tcp [ 2118.308446] Key type .llcrypt unregistered [ 2118.312690] Key type ._llcrypt unregistered [ 2133.237460] Key type ._llcrypt registered [ 2133.240295] Key type .llcrypt registered [ 2134.458315] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 15:51:28 (1788465088) [ 2135.404360] Key type .llcrypt unregistered [ 2135.406423] Key type ._llcrypt unregistered [ 2142.133173] Key type ._llcrypt registered [ 2142.136067] Key type .llcrypt registered [ 2142.459497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2142.486295] alg: No test for adler32 (adler32-zlib) [ 2149.816867] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 15:51:43 (1788465103) [ 2150.190592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.256001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.348408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2150.396540] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2151.086992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.124854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2151.168905] LNet: Removed LNI 192.168.203.33@tcp [ 2151.489183] LNet: 106251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.075549] Key type .llcrypt unregistered [ 2152.079773] Key type ._llcrypt unregistered [ 2164.439546] Key type ._llcrypt registered [ 2164.440763] Key type .llcrypt registered [ 2165.413624] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 15:51:59 (1788465119) [ 2166.072970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2166.104657] alg: No test for adler32 (adler32-zlib) [ 2167.095835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2167.199802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2167.233510] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2168.047335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2168.091280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.33@tcp [ 2168.107890] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2168.118685] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.125886] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 0 [ 2168.142772] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2168.159756] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2168.171779] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.182661] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 0 [ 2168.201965] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2168.214894] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2168.232949] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.247063] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 2218 [ 2168.259143] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2168.263773] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2168.278442] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2168.293486] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2168, 0, 2218 [ 2168.312695] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2176.009252] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 15:52:09 (1788465129) [ 2176.570340] LNet: 107788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2176.575850] LNetError: 107788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2176.591457] LNet: Removed LNI 192.168.203.33@tcp [ 2177.365070] Key type .llcrypt unregistered [ 2177.367763] Key type ._llcrypt unregistered [ 2186.818242] Key type ._llcrypt registered [ 2186.822930] Key type .llcrypt registered [ 2187.034473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.058577] alg: No test for adler32 (adler32-zlib) [ 2187.972065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2188.000880] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2195.647319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 2205.921545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.950717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2205.981184] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2207.173320] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2207.307557] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2207.315946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2207.320528] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2209.577859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2209.606295] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2224.538336] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 15:52:57 (1788465177) [ 2224.855584] LNet: 109290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2224.869720] LNetError: 109290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2224.878898] LNet: Removed LNI 192.168.203.33@tcp [ 2225.888133] LNet: Removed LNI 192.168.203.33@tcp1 [ 2225.892400] LNetError: 109290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2225.895133] LNetError: 109290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2225.932618] LNet: Removed LNI 10.1.2.3@tcp99 [ 2226.769479] Key type .llcrypt unregistered [ 2226.771214] Key type ._llcrypt unregistered [ 2236.194754] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2236.211937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2238.782762] Key type ._llcrypt registered [ 2238.786212] Key type .llcrypt registered [ 2239.096441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2239.114893] alg: No test for adler32 (adler32-zlib) [ 2239.972689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2240.033814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2240.062563] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2241.003842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2241.021433] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2243.245188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2243.296767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2243.329968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2243.349523] LNetError: 110032:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2243.381189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2243.434913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2243.797496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2243.849195] LNet: 110053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2243.855175] LNetError: 110053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2243.863955] LNet: Removed LNI 10.1.2.3@tcp [ 2244.905086] LNet: Removed LNI 192.168.203.33@tcp [ 2245.432504] Key type .llcrypt unregistered [ 2245.439331] Key type ._llcrypt unregistered [ 2259.160951] Key type ._llcrypt registered [ 2259.168081] Key type .llcrypt registered [ 2260.861504] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 15:53:33 (1788465213) [ 2261.336323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2261.357336] alg: No test for adler32 (adler32-zlib) [ 2262.255478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.300911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2262.328990] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2263.128712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2263.209636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg333-client@tcp [ 2270.463783] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 15:53:44 (1788465224) [ 2270.577199] LNet: 111575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2270.588993] LNetError: 111575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.604619] LNet: Removed LNI 192.168.203.33@tcp [ 2271.222848] Key type .llcrypt unregistered [ 2271.227251] Key type ._llcrypt unregistered [ 2277.949831] Key type ._llcrypt registered [ 2277.954094] Key type .llcrypt registered [ 2278.130061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2278.157042] alg: No test for adler32 (adler32-zlib) [ 2279.017934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2279.036536] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2279.066736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2285.558831] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 15:53:59 (1788465239) [ 2287.067878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2289.227179] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 15:54:02 (1788465242) [ 2290.491505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2292.264922] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 15:54:06 (1788465246) [ 2292.614708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.658392] LNet: 113004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2292.671559] LNetError: 113004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.693642] LNet: Removed LNI 192.168.203.33@tcp [ 2292.729040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.821961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2292.847238] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2293.705474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2293.739685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2300.820475] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 15:54:14 (1788465254) [ 2301.273645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.338798] LNet: 113713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2301.360630] LNetError: 113713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.396551] LNet: Removed LNI 192.168.203.33@tcp [ 2301.439695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.536463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2301.600479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2302.123561] Key type .llcrypt unregistered [ 2302.134570] Key type ._llcrypt unregistered [ 2314.255170] Key type ._llcrypt registered [ 2314.260185] Key type .llcrypt registered [ 2316.182934] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 15:54:29 (1788465269) [ 2316.568259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.582729] alg: No test for adler32 (adler32-zlib) [ 2317.552467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.607930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2317.630914] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2318.472180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2318.539478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2325.573990] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 15:54:38 (1788465278) [ 2325.696764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.747665] LNet: 115258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.771321] LNetError: 115258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.802429] LNet: Removed LNI 192.168.203.33@tcp [ 2325.887967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.021724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2326.074685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2332.871328] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 15:54:46 (1788465286) [ 2333.138559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.171929] LNet: 115846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.219449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.318962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2333.862599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2333.897417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2333.966653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2334.475442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.520904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2334.639517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.672437] LNet: 115879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2334.713512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.836243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2335.352958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.432153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2335.532370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2335.883511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.912796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2341.440808] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 15:54:55 (1788465295) [ 2341.705322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.721483] LNet: 116480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.737645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.795601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.842428] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2342.605787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2342.630779] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2343.599328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2343.678476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2343.879800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.906845] LNetError: 116759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.915247] LNet: Removed LNI 192.168.203.33@tcp [ 2343.929097] LNet: Removed LNI 192.168.203.33@tcp2 [ 2343.962085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.045496] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2344.050789] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2344.060859] LNet: 116765:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.667173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2344.889632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.922573] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.929594] LNetError: 116802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2344.935382] LNet: Removed LNI 192.168.203.33@tcp [ 2344.946635] LNet: Removed LNI 192.168.203.33@tcp2 [ 2344.967527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.013948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.047025] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2345.792221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2345.815751] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2346.567817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2346.597877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2346.663596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2346.850277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.874730] LNet: 117087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.880056] LNet: 117087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2347.936485] LNet: Removed LNI 192.168.203.33@tcp [ 2347.942428] LNetError: 117087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2347.957356] LNetError: 117087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2347.969740] LNet: Removed LNI 192.168.203.33@tcp2 [ 2348.003489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.084203] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2348.090569] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2348.094611] LNet: 117093:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.621774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2348.844803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.869977] LNet: Removed LNI 192.168.203.33@tcp [ 2349.926617] LNet: Removed LNI 192.168.203.33@tcp2 [ 2349.958841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.018609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.050843] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2350.816784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2350.842178] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2351.841797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2351.896985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2352.144923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.176482] LNetError: 117410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.189527] LNetError: 117410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2352.199356] LNet: Removed LNI 192.168.203.33@tcp [ 2352.226603] LNet: Removed LNI 192.168.203.33@tcp2 [ 2352.294865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2352.376565] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2352.382490] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2352.388717] LNet: 117416:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2352.981510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2353.205405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2354.272176] LNet: Removed LNI 192.168.203.33@tcp [ 2354.290890] LNet: Removed LNI 192.168.203.33@tcp2 [ 2354.320460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2354.402295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2354.436685] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2355.070317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2355.118527] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2356.150109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2356.185418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2356.246928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2356.468560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.488377] LNet: 117738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.492922] LNet: 117738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2357.537010] LNet: Removed LNI 192.168.203.33@tcp [ 2358.570986] LNet: Removed LNI 192.168.203.33@tcp2 [ 2358.617370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.692186] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2358.697170] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2358.705345] LNet: 117744:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2359.357938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2359.691456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.722454] LNet: Removed LNI 192.168.203.33@tcp [ 2360.743221] LNet: Removed LNI 192.168.203.33@tcp2 [ 2360.768949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.843057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.884731] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2361.535890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2361.565677] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2362.351477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2362.378947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2362.433236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2362.775746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.835207] LNetError: 118078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.843874] LNetError: 118078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2362.852745] LNet: Removed LNI 192.168.203.33@tcp [ 2362.866866] LNet: Removed LNI 192.168.203.33@tcp2 [ 2362.881523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.949589] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2362.954378] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2362.958568] LNet: 118084:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.494235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2363.794861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.832256] LNet: Removed LNI 192.168.203.33@tcp [ 2364.842681] LNet: Removed LNI 192.168.203.33@tcp2 [ 2364.875996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.948765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.996730] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2365.696521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2365.729612] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2366.563737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2366.621952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2366.652606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2366.728848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2367.134426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.224128] LNet: Removed LNI 192.168.203.33@tcp [ 2369.258686] LNet: Removed LNI 192.168.203.33@tcp2 [ 2369.296904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.367431] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2369.375695] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2369.894490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2376.334586] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2377.973781] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 15:55:31 (1788465331) [ 2378.304367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.323499] LNet: 119205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2378.335074] LNet: 119205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2378.345521] LNet: Removed LNI 192.168.203.33@tcp [ 2379.430181] LNet: Removed LNI 192.168.203.33@tcp2 [ 2379.447636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.700946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2379.752864] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2380.462269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2380.480896] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2381.220830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.255667] LNetError: 119478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2381.261797] LNetError: 119478:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2381.267933] LNet: Removed LNI 10.1.2.3@tcp [ 2381.278122] LNet: Removed LNI 192.168.203.33@tcp [ 2381.298753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2381.414896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2381.459069] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2381.464797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2381.870404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2381.901558] LNet: Removed LNI 192.168.203.33@tcp [ 2381.928291] LNet: Removed LNI 10.1.2.3@tcp [ 2387.893188] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 15:55:41 (1788465341) [ 2388.050354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2388.083781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.153883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2388.180914] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2388.840507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2388.870450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2388.904416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2388.932212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2388.957260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.007638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.053217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.089972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.132457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.163150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.205956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.248212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2389.355958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2395.535974] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 15:55:49 (1788465349) [ 2395.706177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.776107] LNet: Removed LNI 192.168.203.33@tcp [ 2396.810654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.898690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.924657] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2397.852231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2397.907453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2398.003879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.108896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.167535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.213592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.266316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.299965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.343858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.400192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.437587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.475480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.565301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2398.630871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.693467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.743753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.802996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.846965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.891976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.942901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2398.986831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2399.037895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2399.080233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.33@tcp [ 2405.276805] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 15:55:59 (1788465359) [ 2405.493814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.573372] LNet: Removed LNI 192.168.203.33@tcp [ 2406.612896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.743108] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2406.843316] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2406.849247] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2406.863220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2409.272759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2409.308058] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2410.003575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2410.038665] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2412.053906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2412.109876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2412.170610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.33@tcp -m PUT -e local_timeout -r 1 [ 2412.261733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2412.314030] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2412.319807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2412.592632] LNetError: 121642:0:(lib-move.c:4794:lnet_parse()) 192.168.203.33@tcp, src 192.168.203.33@tcp: Dropping PUT (error -108 looking up sender) [ 2413.600516] LNet: 122102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.609587] LNet: 122102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2413.635338] LNetError: 122102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2413.651863] LNetError: 122102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2413.666771] LNet: Removed LNI 10.1.2.3@tcp [ 2414.700336] LNet: Removed LNI 192.168.203.33@tcp [ 2415.279887] Key type .llcrypt unregistered [ 2415.285899] Key type ._llcrypt unregistered [ 2426.215650] Key type ._llcrypt registered [ 2426.218889] Key type .llcrypt registered [ 2427.906669] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 15:56:21 (1788465381) [ 2428.474913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2428.500679] alg: No test for adler32 (adler32-zlib) [ 2429.380359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.741705] LNet: 122952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.376337] Key type .llcrypt unregistered [ 2430.380250] Key type ._llcrypt unregistered [ 2436.991177] Key type ._llcrypt registered [ 2436.993632] Key type .llcrypt registered [ 2437.238083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2437.264831] alg: No test for adler32 (adler32-zlib) [ 2438.166637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2438.189562] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2445.827646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 2446.038747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2446.637477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.133@tcp [ 2446.889520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2446.918082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2446.950860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2446.983970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2447.008529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2447.035602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2447.064734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2447.093839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2447.125759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2447.146126] LNet: 123401:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2447.162405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2447.189734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2447.242375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2447.286947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2447.327762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2447.373865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2448.091414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2448.363443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2448.387150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2448.414085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2448.437964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.463481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2448.481154] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 2448.489673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.515526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.541401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.569815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.588681] LNet: 123521:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.604728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2448.633708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2448.661716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2448.695964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2448.728286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2448.767261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2449.386173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.133@tcp [ 2449.617075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2449.651202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2449.696543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2449.738719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2449.796746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2449.832314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2449.859639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2449.890509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2449.911967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2449.932262] LNet: 123640:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2449.947441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2449.975837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2450.025932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2450.064762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2450.116574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2450.179600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.985063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2451.187652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2451.210495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2451.235415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2451.277043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2451.311727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2451.353932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2451.390927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2451.439702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2451.467868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2451.498664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2451.534410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2451.581592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2451.619684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2451.648904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2451.685557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2451.994905] LNet: 123806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2452.010789] LNetError: 123806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.025672] LNet: Removed LNI 192.168.203.33@tcp [ 2452.434661] Key type .llcrypt unregistered [ 2452.438568] Key type ._llcrypt unregistered [ 2470.888945] Key type ._llcrypt registered [ 2470.891773] Key type .llcrypt registered [ 2472.335339] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 15:57:05 (1788465425) [ 2472.714761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2472.745142] alg: No test for adler32 (adler32-zlib) [ 2473.663298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.189418] LNet: 124691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2474.954381] Key type .llcrypt unregistered [ 2474.959775] Key type ._llcrypt unregistered [ 2481.962252] Key type ._llcrypt registered [ 2481.964879] Key type .llcrypt registered [ 2482.218718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2482.256576] alg: No test for adler32 (adler32-zlib) [ 2483.265984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2483.321118] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2491.886427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.133@tcp [ 2503.462686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2503.524997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2503.559164] LNet: Added LNI 192.168.203.33@tcp1 [8/256/0/180] [ 2504.971776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2505.916208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.133@tcp [ 2506.263368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2506.302613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2506.340769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2506.397362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2506.444477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2506.494355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2506.539687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2506.622798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2506.671866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2506.688447] LNet: 125343:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2506.726651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2506.781814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2506.812391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2506.865451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2506.912520] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.133@tcp [ 2506.928984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2506.991733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2507.992624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2508.407581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2508.498383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2508.556195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2508.616882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2508.679418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2508.744790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2508.822265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2508.894621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2508.947658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2508.983547] LNet: 125462:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2509.014028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2509.082431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2509.162095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2509.259262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2509.327703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2509.404686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2510.593959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.133@tcp [ 2511.044633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2511.098224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2511.157500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2511.219616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2511.270037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2511.334043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2511.418866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2511.485707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2511.537160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2511.557258] LNet: 125582:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2511.582828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2511.628981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2511.672740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2511.723671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2511.788715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2511.824615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2512.945378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2513.230379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2513.282673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2513.337672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2513.392376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2513.462513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2513.489485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2513.512677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2513.541548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2513.572694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2513.589331] LNet: 125701:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2513.606773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2513.632565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2513.677428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2513.728452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2513.789943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2513.843324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2521.154766] LNet: 125782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.160587] LNetError: 125782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.167220] LNet: Removed LNI 192.168.203.33@tcp [ 2521.176150] LNet: Removed LNI 192.168.203.33@tcp1 [ 2521.670685] Key type .llcrypt unregistered [ 2521.672634] Key type ._llcrypt unregistered [ 2541.140598] Key type ._llcrypt registered [ 2541.145080] Key type .llcrypt registered [ 2542.623870] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 15:58:16 (1788465496) [ 2543.022789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.044250] alg: No test for adler32 (adler32-zlib) [ 2543.977840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2544.038642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2544.058458] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2544.966974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2544.988363] LNet: Added LNI 192.168.203.33@tcp2 [8/256/0/180] [ 2545.838363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.33@tcp [ 2545.929767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2546.976163] LNet: Removed LNI 192.168.203.33@tcp [ 2553.244813] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 15:58:27 (1788465507) [ 2554.841767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2556.418969] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 15:58:30 (1788465510) [ 2563.952773] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 15:58:37 (1788465517) [ 2572.298521] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 15:58:45 (1788465525) [ 2582.275792] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 15:58:55 (1788465535) [ 2591.175097] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 15:59:04 (1788465544) [ 2600.619935] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 15:59:14 (1788465554) [ 2608.876082] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 15:59:22 (1788465562) [ 2615.436427] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 15:59:29 (1788465569) [ 2622.372902] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 15:59:35 (1788465575) [ 2630.122208] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 15:59:43 (1788465583) [ 2637.436438] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 15:59:51 (1788465591) [ 2645.357217] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 15:59:59 (1788465599) [ 2652.433157] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 16:00:06 (1788465606) [ 2659.381488] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 16:00:13 (1788465613) [ 2666.774511] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 16:00:20 (1788465620) [ 2673.963658] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 16:00:27 (1788465627) [ 2682.271660] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 16:00:35 (1788465635) [ 2690.137257] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 16:00:43 (1788465643) [ 2690.253539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2690.271073] LNet: 137469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2690.288499] LNetError: 137469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2690.311933] LNet: Removed LNI 192.168.203.33@tcp2 [ 2690.326810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2690.428645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2690.481919] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2691.556870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2691.591793] LNet: 137614:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2691.608802] LNetError: 137614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2691.634178] LNet: Removed LNI 192.168.203.33@tcp [ 2691.658507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2691.697219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2691.717492] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2699.140809] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 16:00:52 (1788465652) [ 2705.737800] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 16:00:59 (1788465659) [ 2712.508300] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 16:01:06 (1788465666) [ 2718.872879] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 16:01:12 (1788465672) [ 2724.438491] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 16:01:18 (1788465678) [ 2729.841231] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 16:01:23 (1788465683) [ 2735.636599] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 16:01:29 (1788465689) [ 2742.505333] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 16:01:36 (1788465696) [ 2748.679807] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 16:01:42 (1788465702) [ 2755.441915] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 16:01:49 (1788465709) [ 2762.190586] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 16:01:55 (1788465715) [ 2769.160424] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 16:02:02 (1788465722) [ 2775.423870] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2474 sec ========== 16:02:09 (1788465729) [ 2775.723950] LNet: 145092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2775.729838] LNetError: 145092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2775.743496] LNet: Removed LNI 192.168.203.33@tcp [ 2776.264534] Key type .llcrypt unregistered [ 2776.267721] Key type ._llcrypt unregistered [ 2786.437627] Key type ._llcrypt registered [ 2786.441735] Key type .llcrypt registered [ 2786.821931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2786.836663] alg: No test for adler32 (adler32-zlib) [ 2787.920568] Lustre: Lustre: Build Version: 2.17.58_2_gbb63b80 [ 2788.153113] LNet: Added LNI 192.168.203.33@tcp [8/256/0/180] [ 2789.808183] Key type lgssc registered [ 2790.992529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2850.103283] Lustre: Mounted lustre-client - version 2.17.58_2_gbb63b80 [ 2854.754818] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2861.821127] Lustre: Unmounted lustre-client [ 2885.146768] Key type lgssc unregistered [ 2885.609745] LNet: 148039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2885.622768] LNetError: 148039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2885.651848] LNet: Removed LNI 192.168.203.33@tcp [ 2886.857217] Key type .llcrypt unregistered [ 2886.859533] Key type ._llcrypt unregistered