[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 449390345 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002082] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004020] kvm-guest: setup PV IPIs [ 0.007621] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010007] pid_max: default: 32768 minimum: 301 [ 0.011117] LSM: Security Framework initializing [ 0.012041] Yama: becoming mindful. [ 0.013031] SELinux: Initializing. [ 0.014058] *** VALIDATE selinux *** [ 0.022758] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026607] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027138] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028093] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029110] *** VALIDATE tmpfs *** [ 0.031404] *** VALIDATE proc *** [ 0.032208] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034244] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036030] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039023] Spectre V2 : User space: Vulnerable [ 0.040019] Speculative Store Bypass: Vulnerable [ 0.042699] debug: unmapping init [mem 0xffffffffb4e59000-0xffffffffb4e60fff] [ 0.044174] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045667] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046024] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050012] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.053265] rcu: Hierarchical SRCU implementation. [ 0.055533] smp: Bringing up secondary CPUs ... [ 0.056626] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.064071] smp: Brought up 1 node, 4 CPUs [ 0.066008] smpboot: Max logical packages: 1 [ 0.067015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.108865] node 0 deferred pages initialised in 39ms [ 0.111504] devtmpfs: initialized [ 0.113269] x86/mm: Memory block size: 128MB [ 0.116502] gcov: version magic: 0x41383552 [ 0.118221] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.119073] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.120198] pinctrl core: initialized pinctrl subsystem [ 0.121155] [ 0.121667] ************************************************************* [ 0.122008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123006] ** ** [ 0.124010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125014] ** ** [ 0.126009] ** This means that this kernel is built to expose internal ** [ 0.127008] ** IOMMU data structures, which may compromise security on ** [ 0.128008] ** your system. ** [ 0.129007] ** ** [ 0.130006] ** If you see this message and you are not debugging the ** [ 0.131008] ** kernel, report this immediately to your vendor! ** [ 0.132012] ** ** [ 0.133008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.134008] ************************************************************* [ 0.135494] NET: Registered protocol family 16 [ 0.136400] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.137037] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.138035] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.139534] cpuidle: using governor menu [ 0.140829] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.143534] PCI: Using configuration type 1 for base access [ 0.145127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.153216] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.155025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.158135] cryptd: max_cpu_qlen set to 1000 [ 0.161247] ACPI: Added _OSI(Module Device) [ 0.163012] ACPI: Added _OSI(Processor Device) [ 0.164018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165008] ACPI: Added _OSI(Processor Aggregator Device) [ 0.170515] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176121] ACPI: Interpreter enabled [ 0.177114] ACPI: PM: (supports S0 S3 S4 S5) [ 0.179011] ACPI: Using IOAPIC for interrupt routing [ 0.180179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.183458] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.197916] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.200042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.203016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.206072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.211648] acpiphp: Slot [2] registered [ 0.213127] acpiphp: Slot [5] registered [ 0.215187] acpiphp: Slot [6] registered [ 0.217152] acpiphp: Slot [3] registered [ 0.219079] acpiphp: Slot [4] registered [ 0.221093] acpiphp: Slot [7] registered [ 0.223101] acpiphp: Slot [8] registered [ 0.224201] acpiphp: Slot [9] registered [ 0.226077] acpiphp: Slot [10] registered [ 0.228160] acpiphp: Slot [11] registered [ 0.229231] acpiphp: Slot [12] registered [ 0.231290] acpiphp: Slot [13] registered [ 0.233092] acpiphp: Slot [14] registered [ 0.235103] acpiphp: Slot [15] registered [ 0.237099] acpiphp: Slot [16] registered [ 0.239109] acpiphp: Slot [17] registered [ 0.241096] acpiphp: Slot [18] registered [ 0.243115] acpiphp: Slot [19] registered [ 0.245138] acpiphp: Slot [20] registered [ 0.247119] acpiphp: Slot [21] registered [ 0.249099] acpiphp: Slot [22] registered [ 0.251097] acpiphp: Slot [23] registered [ 0.253121] acpiphp: Slot [24] registered [ 0.255165] acpiphp: Slot [25] registered [ 0.257190] acpiphp: Slot [26] registered [ 0.259117] acpiphp: Slot [27] registered [ 0.261102] acpiphp: Slot [28] registered [ 0.263113] acpiphp: Slot [29] registered [ 0.265116] acpiphp: Slot [30] registered [ 0.267173] acpiphp: Slot [31] registered [ 0.270061] PCI host bridge to bus 0000:00 [ 0.272016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.275023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.277020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.280024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283110] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.292000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.292000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.300013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.306973] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.309017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.311017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.315018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.317624] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.320961] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.324037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.327950] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.344994] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.349975] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.357087] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.395023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.423012] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.435013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.448011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.457668] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.460380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.463820] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.465424] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.466172] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.469047] iommu: Default domain type: Passthrough [ 0.471210] SCSI subsystem initialized [ 0.472128] ACPI: bus type USB registered [ 0.473096] usbcore: registered new interface driver usbfs [ 0.476098] usbcore: registered new interface driver hub [ 0.478073] usbcore: registered new device driver usb [ 0.480148] pps_core: LinuxPPS API ver. 1 registered [ 0.482010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.484054] PTP clock support registered [ 0.487093] EDAC MC: Ver: 3.0.0 [ 0.489112] PCI: Using ACPI for IRQ routing [ 0.492961] NetLabel: Initializing [ 0.494019] NetLabel: domain hash size = 128 [ 0.495010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.498087] NetLabel: unlabeled traffic allowed by default [ 0.500123] vgaarb: loaded [ 0.501311] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.503015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.512287] clocksource: Switched to clocksource kvm-clock [ 0.633862] VFS: Disk quotas dquot_6.6.0 [ 0.635514] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.638384] *** VALIDATE ramfs *** [ 0.639789] *** VALIDATE hugetlbfs *** [ 0.641183] pnp: PnP ACPI init [ 0.643435] pnp: PnP ACPI: found 6 devices [ 0.661397] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.664623] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.666688] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.668732] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.670740] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.672887] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.675528] NET: Registered protocol family 2 [ 0.678163] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.682455] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.685717] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.690558] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.693764] TCP: Hash tables configured (established 65536 bind 65536) [ 0.696644] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.699881] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.702506] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.705340] NET: Registered protocol family 1 [ 0.891924] RPC: Registered named UNIX socket transport module. [ 0.906824] RPC: Registered udp transport module. [ 0.909606] RPC: Registered tcp transport module. [ 0.913238] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.919025] NET: Registered protocol family 44 [ 0.923539] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.927082] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.930241] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.934130] PCI: CLS 0 bytes, default 64 [ 0.954663] Unpacking initramfs... [ 4.117451] debug: unmapping init [mem 0xffff96d57cc64000-0xffff96d57ffcffff] [ 4.127337] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.130750] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.134522] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.539622] Initialise system trusted keyrings [ 5.541142] Key type blacklist registered [ 5.542922] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.552356] zbud: loaded [ 5.555328] *** VALIDATE nfs *** [ 5.556410] *** VALIDATE nfs4 *** [ 5.558150] pstore: using deflate compression [ 5.561655] Platform Keyring initialized [ 5.716290] NET: Registered protocol family 38 [ 5.718343] Key type asymmetric registered [ 5.720584] Asymmetric key parser 'x509' registered [ 5.722896] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 5.729458] io scheduler mq-deadline registered [ 5.731357] io scheduler kyber registered [ 5.739590] io scheduler bfq registered [ 5.741815] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 5.746472] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 5.758430] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 5.767227] ACPI: Power Button [PWRF] [ 5.778417] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.789307] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.804714] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.849865] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.894190] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.906815] Non-volatile memory driver v1.3 [ 5.911264] Linux agpgart interface v0.103 [ 6.011817] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 6.016068] vda: detected capacity change from 0 to 75096064 [ 6.057259] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.063504] vdb: detected capacity change from 0 to 1073741824 [ 6.075574] libphy: Fixed MDIO Bus: probed [ 6.089766] usbcore: registered new interface driver usbserial_generic [ 6.094697] usbserial: USB Serial support registered for generic [ 6.099533] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.108543] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.112561] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.118095] mousedev: PS/2 mouse device common for all mice [ 6.122721] rtc_cmos 00:05: RTC can wake from S4 [ 6.128192] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.130804] rtc_cmos 00:05: registered as rtc0 [ 6.140117] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.142026] intel_pstate: CPU model not supported [ 6.154681] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.161216] hid: raw HID events driver (C) Jiri Kosina [ 6.167643] usbcore: registered new interface driver usbhid [ 6.169171] usbhid: USB HID core driver [ 6.172457] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.176684] drop_monitor: Initializing network drop monitor service [ 6.176809] Initializing XFRM netlink socket [ 6.177497] NET: Registered protocol family 10 [ 6.187436] Segment Routing with IPv6 [ 6.206106] NET: Registered protocol family 17 [ 6.219192] mpls_gso: MPLS GSO support [ 6.231796] RAS: Correctable Errors collector initialized. [ 6.236307] AVX version of gcm_enc/dec engaged. [ 6.240237] AES CTR mode by8 optimization enabled [ 6.611908] sched_clock: Marking stable (6611888753, 0)->(7471011913, -859123160) [ 6.629930] registered taskstats version 1 [ 6.638906] Loading compiled-in X.509 certificates [ 6.645227] zswap: loaded using pool lzo/zbud [ 6.800388] Key type big_key registered [ 6.853365] Key type encrypted registered [ 6.854853] ima: No TPM chip found, activating TPM-bypass! [ 6.861524] ima: Allocated hash algorithm: sha1 [ 6.865782] ima: No architecture policies found [ 6.881147] evm: Initialising EVM extended attributes: [ 6.892908] evm: security.selinux [ 6.898355] evm: security.ima [ 6.901378] evm: security.capability [ 6.908966] evm: HMAC attrs: 0x1 [ 6.939809] rtc_cmos 00:05: setting system clock to 2026-09-13 02:02:38 UTC (1789264958) [ 6.966797] debug: unmapping init [mem 0xffffffffb5e03000-0xffffffffb5ffffff] [ 6.987920] debug: unmapping init [mem 0xffffffffb4b82000-0xffffffffb4e58fff] [ 7.000116] Write protecting the kernel read-only data: 28672k [ 7.029961] debug: unmapping init [mem 0xffffffffb3203000-0xffffffffb33fffff] [ 7.055098] debug: unmapping init [mem 0xffffffffb3b14000-0xffffffffb3bfffff] [ 7.206479] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.217991] systemd[1]: Detected virtualization kvm. [ 7.228129] systemd[1]: Detected architecture x86-64. [ 7.235882] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.324867] systemd[1]: No hostname configured. [ 7.327461] systemd[1]: Set hostname to . [ 7.329976] random: systemd: uninitialized urandom read (16 bytes read) [ 7.333472] systemd[1]: Initializing machine ID from random generator. [ 7.470627] random: ln: uninitialized urandom read (6 bytes read) [ 7.843814] random: systemd: uninitialized urandom read (16 bytes read) [ 7.864624] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.899665] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 7.978742] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.945791] device-mapper: uevent: version 1.0.3 [ 9.948219] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK [0[ 10.725914] hrtimer: interrupt took 6853318 ns m] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 12.861633] virtio_net virtio0 ens2: renamed from eth0 [ 13.390533] scsi host0: ata_piix [ 13.423174] scsi host1: ata_piix [ 13.441152] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 13.459029] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 17.723241] random: fast init done [ 18.464613] random: crng init done [ 18.465758] random: 7 urandom warning(s) missed due to ratelimiting [ 21.004697] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 24.968837] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 30.155495] printk: systemd: 26 output lines suppressed due to ratelimiting [ 31.867653] SELinux: Disabled at runtime. [ 32.220366] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.252650] systemd[1]: Detected virtualization kvm. [ 32.254380] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 37.435788] systemd[1]: initrd-switch-root.service: Succeeded. [ 37.451213] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 37.472883] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 37.489609] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 37.613119] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 37.649189] systemd[1]: Starting Journal Service... Starting Journal Service... [ 37.681083] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ 37.696490] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ 37.714780] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 37.722907] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ 39.629371] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 41.906342] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 43.751382] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 43.961779] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.795938] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 45.049284] EDAC sbridge: Ver: 1.1.2 [ 47.635519] Key type dns_resolver registered [ 48.413563] NFS: Registering the id_resolver key type [ 48.446954] Key type id_resolver registered [ 48.448437] Key type id_legacy registered [* ] A start job is running for Configur…only root support (11s / no limit) [** ] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started irqbalance daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg159-client login: [ 137.076949] libcfs: loading out-of-tree module taints kernel. [ 137.509469] Key type ._llcrypt registered [ 137.513300] Key type .llcrypt registered [ 138.325814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 138.373284] alg: No test for adler32 (adler32-zlib) [ 140.119414] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 141.424674] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 143.288402] Key type lgssc registered [ 145.471985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 269.120176] Lustre: Mounted lustre-client - version 2.17.57_105_g2595dd0 [ 274.463944] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 285.722206] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing check_logdir /tmp/testlogs/ [ 291.529329] Lustre: DEBUG MARKER: oleg159-client.virtnet: executing yml_node [ 294.879416] Lustre: lustre-OST0000-osc-ffff96d5c6303000: disconnect after 23s idle [ 295.845575] Lustre: DEBUG MARKER: Client: 2.17.57.105 [ 298.283269] Lustre: DEBUG MARKER: MDS: 2.17.57.105 [ 300.705566] Lustre: DEBUG MARKER: OSS: 2.17.57.105 [ 302.010363] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 12 22:07:32 EDT 2026 [ 316.866877] Lustre: Unmounted lustre-client [ 335.740689] Key type lgssc unregistered [ 336.068305] LNet: 8402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.075896] LNetError: 8402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 337.149330] LNet: Removed LNI 192.168.201.59@tcp [ 338.006250] Key type .llcrypt unregistered [ 338.011775] Key type ._llcrypt unregistered [ 363.088199] Key type ._llcrypt registered [ 363.093859] Key type .llcrypt registered [ 363.279188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 363.301311] alg: No test for adler32 (adler32-zlib) [ 364.199218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 364.280361] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 364.303358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 367.721989] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 367.892757] LNet: 9281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.895434] LNetError: 9281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 367.919103] LNet: Removed LNI 192.168.201.59@tcp [ 368.445995] Key type .llcrypt unregistered [ 368.447927] Key type ._llcrypt unregistered [ 373.428257] Key type ._llcrypt registered [ 373.429970] Key type .llcrypt registered [ 374.389526] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 22:08:44 (1789265324) [ 374.779135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 374.790253] alg: No test for adler32 (adler32-zlib) [ 375.675807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.781960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.808302] LNet: 9688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.826986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.889678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 375.903348] LNet: 9695:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 381.154777] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 22:08:51 (1789265331) [ 381.399894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.427860] LNet: 10274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.455822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.555553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 387.532337] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 22:08:57 (1789265337) [ 387.688606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.719229] LNet: 10855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.756232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.867884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 394.031665] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 22:09:04 (1789265344) [ 394.131622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.168578] LNet: 11439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.200519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.321222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 401.596593] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 22:09:11 (1789265351) [ 401.986921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.013037] LNet: 12032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.031242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.108207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 402.610833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 410.771395] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 22:09:20 (1789265360) [ 411.212481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.242619] LNet: 12625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 411.278618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.437443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 411.999843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 412.678817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 421.851379] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 22:09:31 (1789265371) [ 421.964562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.023420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.172123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 430.884213] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 22:09:40 (1789265380) [ 431.021181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.061642] LNet: 13800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.072098] LNet: 13800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 431.102749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.233045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 431.263982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.622445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 431.662390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 432.247661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 432.321519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.834977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 432.876611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 433.511369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 433.561563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 434.072452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 434.143283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 434.727080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 434.770186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.152759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 435.201727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 435.622472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 435.662431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 436.096706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 436.120737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 436.728437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 436.798594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 437.265272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 437.310401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 437.787693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 437.824364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 445.146793] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 22:09:55 (1789265395) [ 445.233446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.261760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.327091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 445.346553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 452.338874] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 22:10:02 (1789265402) [ 452.694590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.766324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.951102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 453.020376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 462.557437] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 22:10:12 (1789265412) [ 462.827661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.866072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.935953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 462.965517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 470.677529] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 22:10:20 (1789265420) [ 471.171540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.211109] LNet: 16249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 471.222576] LNet: 16249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 471.262950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.446405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 471.526489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 479.638455] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 22:10:29 (1789265429) [ 479.721524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.771494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.888384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 479.911443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 487.514878] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 22:10:37 (1789265437) [ 487.673618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.748206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.844530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.905851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 495.197410] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 22:10:45 (1789265445) [ 495.352634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.439609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.608396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.188902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.980806] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 22:10:54 (1789265454) [ 504.268352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.349910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.660420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.953849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.366665] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 22:11:01 (1789265461) [ 511.536053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.621324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.772703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.971903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.489207] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 22:11:09 (1789265469) [ 519.692376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.780394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.122541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.314701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 526.883792] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 22:11:16 (1789265476) [ 527.216593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.266944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.347406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 527.383814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 533.587425] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 22:11:23 (1789265483) [ 533.759353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.813933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.888141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 533.939974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.743908] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 22:11:30 (1789265490) [ 541.180900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.208119] LNet: 21690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 541.213914] LNet: 21690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 541.234310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.326214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 547.683786] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 22:11:37 (1789265497) [ 547.894946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.943688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.097727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 554.381073] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 22:11:44 (1789265504) [ 554.436557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.476140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.529669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 554.872293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 561.010608] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 22:11:51 (1789265511) [ 561.210992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.243690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.307141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 561.347099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 568.048667] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 22:11:57 (1789265517) [ 568.139686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.228411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.308240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 568.344586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 575.840433] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 22:12:05 (1789265525) [ 575.939107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.996600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.085344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 576.114498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 582.707670] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 22:12:13 (1789265533) [ 582.863184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 582.925489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.042850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 583.065439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 589.968719] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 22:12:19 (1789265539) [ 590.098962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.184372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.296708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 590.391665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 590.409982] LNetError: 25805:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 590.448729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 590.506594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 599.170370] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 22:12:28 (1789265548) [ 599.353325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.427935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.597135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.uf4SbXxsDX [ 599.721345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.uf4SbXxsDX [ 599.815951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.uf4SbXxsDX [ 608.657326] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 22:12:38 (1789265558) [ 608.785100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.873294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.945262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 608.977817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 609.004878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 614.398150] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 22:12:44 (1789265564) [ 614.635827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.671885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.728946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 614.748748] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 616.158340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 617.215081] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 619.119374] Key type lgssc registered [ 620.300716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 624.052490] LNetError: 27567:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.59@tcp99 has no available nets [ 627.543440] Key type lgssc unregistered [ 627.914462] LNetError: 28671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.936157] LNet: Removed LNI 192.168.201.59@tcp [ 628.701386] Key type .llcrypt unregistered [ 628.709933] Key type ._llcrypt unregistered [ 635.929527] Key type ._llcrypt registered [ 635.931530] Key type .llcrypt registered [ 637.180569] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 22:13:07 (1789265587) [ 637.801198] Key type .llcrypt unregistered [ 637.803132] Key type ._llcrypt unregistered [ 645.532260] Key type ._llcrypt registered [ 645.541676] Key type .llcrypt registered [ 645.851967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.894270] alg: No test for adler32 (adler32-zlib) [ 646.907620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.006219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 647.035046] LNet: 29580:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 647.042159] LNet: 29580:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 647.051631] LNet: 29580:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 647.071643] LNet: Message forwarding will be enabled in 2s [ 650.143118] LNet: Message forwarding enabled [ 650.874505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 650.891519] LNet: Message forwarding will stop in 2s to 6s [ 654.245358] LNet: Message forwarding disabled [ 654.411200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 654.463379] LNet: 29669:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 654.478808] LNet: 29669:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 654.487646] LNet: 29669:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 654.503398] LNet: Message forwarding will be enabled in 2s [ 654.664553] LNet: 29682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.386460] Key type .llcrypt unregistered [ 655.388165] Key type ._llcrypt unregistered [ 669.894606] Key type ._llcrypt registered [ 669.907832] Key type .llcrypt registered [ 671.385895] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 22:13:41 (1789265621) [ 672.008436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.024360] alg: No test for adler32 (adler32-zlib) [ 672.984649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.120815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 680.708473] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 22:13:50 (1789265630) [ 680.816725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.857818] LNet: 31075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.887098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.988892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 681.054188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 681.090952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 681.131554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 681.182810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 681.215782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 681.244784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 681.284896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 681.358988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 681.499929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 681.546906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 681.587145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 681.637922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 681.722867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 681.818060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 681.867369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 681.935678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 681.973210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 682.012884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 682.041089] LNetError: 31125:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 689.546297] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 22:13:59 (1789265639) [ 689.694411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.740238] LNet: 31689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.781500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.920957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.295628] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 22:14:07 (1789265647) [ 699.020316] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg159-server [ 701.652357] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 22:14:11 (1789265651) [ 701.771125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.807510] LNet: 32623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.856935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.950945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.017792] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 703.322495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[60-64]@tcp [ 703.347664] LNet: 32763:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 703.406752] LNetError: 32631:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 703.419264] LNetError: 32631:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.60@tcp has gone from up to down [ 706.463671] LNetError: 32639:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.62@tcp has gone from up to down [ 713.094088] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 22:14:22 (1789265662) [ 715.250567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg159-server [ 717.786618] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 22:14:27 (1789265667) [ 717.892524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.963269] LNet: 33693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.969571] LNetError: 33693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.989399] LNet: Removed LNI 192.168.201.59@tcp [ 718.036618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.186915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.224980] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 719.169545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 719.207764] LNet: 33817:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 719.217825] LNet: 33817:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 719.261594] LNetError: 33703:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.60@tcp is being used as a gateway but routing feature is not turned on [ 719.276120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[60-65/2]@tcp [ 719.278589] LNetError: 33703:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 719.278627] LNetError: 33703:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.60@tcp has gone from up to down [ 719.278632] LNetError: 33703:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 728.090462] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 22:14:37 (1789265677) [ 728.214645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.249614] LNet: 34402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.263547] LNetError: 34402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.291357] LNet: Removed LNI 192.168.201.59@tcp [ 728.331245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.486183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 728.528216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.558185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.643623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.732673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.770666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.824741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 728.898294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.948579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.024688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.129955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.170206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.264310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 729.370602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.425910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.596440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.724768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.761918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.820285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 729.911654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.947560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.055092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.182401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.257729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.330587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 730.434678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.499690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.583983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.730610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.803319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.906398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 731.000882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 731.070832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.201539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 738.771784] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 22:14:48 (1789265688) [ 740.698872] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg159-server [ 742.621966] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 22:14:52 (1789265692) [ 744.834777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg159-server [ 746.653836] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 22:14:56 (1789265696) [ 746.785624] LNet: 35804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.794022] LNet: 35804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 747.740355] Key type .llcrypt unregistered [ 747.747555] Key type ._llcrypt unregistered [ 755.281489] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 755.431623] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 755.437517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 755.440680] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.360926] Key type ._llcrypt registered [ 758.365179] Key type .llcrypt registered [ 758.585941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 758.610803] alg: No test for adler32 (adler32-zlib) [ 759.583758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.704643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 759.740375] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 760.765173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.791394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.086812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 764.107088] LNet: Removed LNI 10.1.2.3@tcp [ 764.245699] LNet: 36526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.259657] LNetError: 36526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.274248] LNet: Removed LNI 192.168.201.59@tcp [ 764.871867] Key type .llcrypt unregistered [ 764.880327] Key type ._llcrypt unregistered [ 782.907784] Key type ._llcrypt registered [ 782.920260] Key type .llcrypt registered [ 784.443654] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 22:15:34 (1789265734) [ 785.035350] Key type .llcrypt unregistered [ 785.037096] Key type ._llcrypt unregistered [ 790.951694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 791.274266] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 791.286921] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 791.292904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 793.998162] Key type ._llcrypt registered [ 794.001690] Key type .llcrypt registered [ 794.424972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.467845] alg: No test for adler32 (adler32-zlib) [ 795.374993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.466261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.488044] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 796.329583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 796.345973] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.360786] LNet: 38009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.366643] LNetError: 38009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 799.372855] LNet: Removed LNI 10.1.2.3@tcp [ 799.384240] LNet: Removed LNI 192.168.201.59@tcp [ 800.238600] Key type .llcrypt unregistered [ 800.249954] Key type ._llcrypt unregistered [ 816.846327] Key type ._llcrypt registered [ 816.850363] Key type .llcrypt registered [ 818.446875] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 22:16:08 (1789265768) [ 819.405479] Key type .llcrypt unregistered [ 819.411354] Key type ._llcrypt unregistered [ 827.748904] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 827.790817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 827.793267] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 827.795816] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 830.550709] Key type ._llcrypt registered [ 830.553286] Key type .llcrypt registered [ 830.832182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.877241] alg: No test for adler32 (adler32-zlib) [ 831.847705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.904740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 831.944573] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 834.032942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 834.068530] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.975642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 837.014783] LNet: Removed LNI 10.1.2.3@tcp [ 837.031895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 837.064302] LNet: Removed LNI 10.1.2.31@tcp [ 837.094342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 837.124563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 838.033600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 838.063467] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 839.028526] LNet: 39921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.033929] LNetError: 39921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 839.038885] LNet: Removed LNI 10.1.2.31@tcp [ 839.052727] LNet: Removed LNI 10.1.2.3@tcp [ 839.818547] Key type .llcrypt unregistered [ 839.822625] Key type ._llcrypt unregistered [ 853.192788] Key type ._llcrypt registered [ 853.195760] Key type .llcrypt registered [ 854.644617] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 22:16:44 (1789265804) [ 855.221027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.250968] alg: No test for adler32 (adler32-zlib) [ 856.168189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.266613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.302874] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 857.256413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.272972] LNet: 40871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.277557] LNetError: 40871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.292348] LNet: Removed LNI 192.168.201.59@tcp [ 857.316852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.393209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.59@tcp [ 857.434061] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 858.428234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.59@tcp [ 858.521890] LNet: Removed LNI 192.168.201.59@tcp [ 858.634512] LNet: 41006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.721553] Key type .llcrypt unregistered [ 859.724639] Key type ._llcrypt unregistered [ 873.242708] Key type ._llcrypt registered [ 873.244718] Key type .llcrypt registered [ 874.482462] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 22:17:04 (1789265824) [ 876.599285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg159-server [ 878.355416] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 22:17:08 (1789265828) [ 878.998447] Key type .llcrypt unregistered [ 879.005856] Key type ._llcrypt unregistered [ 886.547798] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 886.812605] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 886.822743] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 886.836398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 889.473881] Key type ._llcrypt registered [ 889.481050] Key type .llcrypt registered [ 889.788147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 889.819336] alg: No test for adler32 (adler32-zlib) [ 890.756987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.836810] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 891.641924] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 893.760171] LNet: 42838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.765303] LNetError: 42838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.771351] LNet: Removed LNI 192.168.201.59@tcp1 [ 893.786273] LNet: Removed LNI 10.1.2.3@tcp2 [ 894.405665] Key type .llcrypt unregistered [ 894.413286] Key type ._llcrypt unregistered [ 907.962842] Key type ._llcrypt registered [ 907.970365] Key type .llcrypt registered [ 909.507171] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 22:17:39 (1789265859) [ 909.896490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 909.911234] alg: No test for adler32 (adler32-zlib) [ 910.827101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.876454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 910.899631] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 911.720454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.742978] LNet: 43808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.750105] LNetError: 43808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.771786] LNet: Removed LNI 192.168.201.59@tcp [ 911.789939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.858643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 911.880358] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 919.502426] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 22:17:49 (1789265869) [ 919.878482] LNet: 44399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.889252] LNetError: 44399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.914660] LNet: Removed LNI 192.168.201.59@tcp [ 920.516446] Key type .llcrypt unregistered [ 920.518337] Key type ._llcrypt unregistered [ 928.349634] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.584815] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.587122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.600194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.478331] Key type ._llcrypt registered [ 931.480521] Key type .llcrypt registered [ 931.761917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.788228] alg: No test for adler32 (adler32-zlib) [ 932.796296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.911602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.960477] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 933.570651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 933.585762] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 935.739296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.756479] LNet: 45137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.761958] LNetError: 45137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 935.765224] LNet: Removed LNI 10.1.2.3@tcp [ 935.776434] LNet: Removed LNI 192.168.201.59@tcp [ 935.803687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.892479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 935.926652] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 935.929941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 936.589225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.618879] LNet: 45164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.627595] LNetError: 45164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 936.629661] LNetError: 45164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 936.638127] LNet: Removed LNI 10.1.2.3@tcp [ 936.647331] LNet: Removed LNI 192.168.201.59@tcp [ 936.678051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.832334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 936.872649] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 936.876788] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.527340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 937.562523] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 938.482656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.506903] LNet: 45327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.514869] LNetError: 45327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.518276] LNetError: 45327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 938.533446] LNet: Removed LNI 10.1.2.3@tcp [ 938.537611] LNet: Removed LNI 192.168.201.59@tcp [ 938.554210] LNet: Removed LNI 192.168.201.59@tcp2 [ 938.589385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.673285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 938.708682] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 938.716450] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 938.724976] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 946.515959] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 22:18:16 (1789265896) [ 946.650226] LNet: 45916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.661509] LNetError: 45916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 946.671151] LNetError: 45916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 946.678475] LNet: Removed LNI 10.1.2.3@tcp [ 946.680853] LNet: Removed LNI 192.168.201.59@tcp [ 946.704496] LNet: Removed LNI 192.168.201.59@tcp2 [ 947.522389] Key type .llcrypt unregistered [ 947.523918] Key type ._llcrypt unregistered [ 954.496602] Key type ._llcrypt registered [ 954.501691] Key type .llcrypt registered [ 954.774743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.790508] alg: No test for adler32 (adler32-zlib) [ 955.750951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.802463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.822851] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 956.640618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.661383] LNet: 46316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.666805] LNetError: 46316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.679339] LNet: Removed LNI 192.168.201.59@tcp [ 956.699685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.767936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 956.788824] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 957.200371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.218667] LNet: 46346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.226348] LNetError: 46346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.237862] LNet: Removed LNI 192.168.201.59@tcp [ 957.255378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.324744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 957.350505] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 957.735599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.759186] LNet: Removed LNI 192.168.201.59@tcp [ 957.774745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.840949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 957.879085] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 958.542166] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.720744] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.733246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.745703] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.234627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.261851] LNet: 46455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.267224] LNet: 46455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 961.273608] LNetError: 46455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.283061] LNetError: 46455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 961.299121] LNet: Removed LNI 192.168.201.59@tcp [ 961.338304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 961.430446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 961.464331] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 969.201756] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 22:18:38 (1789265918) [ 969.318355] LNet: 47043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.323603] LNetError: 47043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.348033] LNet: Removed LNI 192.168.201.59@tcp [ 970.176193] Key type .llcrypt unregistered [ 970.178116] Key type ._llcrypt unregistered [ 977.206653] Key type ._llcrypt registered [ 977.210439] Key type .llcrypt registered [ 977.463422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 977.478626] alg: No test for adler32 (adler32-zlib) [ 978.350615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.400278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.420218] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 979.380413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.415657] LNet: 47447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.422727] LNetError: 47447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.444698] LNet: Removed LNI 192.168.201.59@tcp [ 979.473961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.615858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 979.663732] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 986.634700] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 22:18:56 (1789265936) [ 986.794200] LNet: 48034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.801394] LNetError: 48034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.829788] LNet: Removed LNI 192.168.201.59@tcp [ 987.351829] Key type .llcrypt unregistered [ 987.354764] Key type ._llcrypt unregistered [ 993.962125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.085581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.093397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.103646] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.719994] Key type ._llcrypt registered [ 996.722900] Key type .llcrypt registered [ 996.902972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 996.919632] alg: No test for adler32 (adler32-zlib) [ 997.905768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.002473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 998.046048] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 998.789659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 998.811210] LNet: 48487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.819936] LNetError: 48487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.846421] LNet: Removed LNI 192.168.201.59@tcp [ 998.867636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.926879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 998.941782] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1005.549385] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 22:19:15 (1789265955) [ 1005.735929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1005.755166] LNet: 49086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.763304] LNetError: 49086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.783397] LNet: Removed LNI 192.168.201.59@tcp [ 1005.807307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1005.898929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1011.990856] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 22:19:22 (1789265962) [ 1012.217661] LNet: 49670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.911103] Key type .llcrypt unregistered [ 1012.912591] Key type ._llcrypt unregistered [ 1019.445264] Key type ._llcrypt registered [ 1019.448618] Key type .llcrypt registered [ 1019.689965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1019.709250] alg: No test for adler32 (adler32-zlib) [ 1020.747935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1020.839336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1020.861103] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1022.093886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1022.125192] LNet: 50063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1022.127857] LNetError: 50063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1022.157220] LNet: Removed LNI 192.168.201.59@tcp [ 1022.218110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1022.306249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1022.336131] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1029.084282] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 22:19:39 (1789265979) [ 1029.223058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1029.265974] LNet: 50649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.274913] LNetError: 50649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.295700] LNet: Removed LNI 192.168.201.59@tcp [ 1029.321077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.425889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1035.240371] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 22:19:45 (1789265985) [ 1036.751622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1038.180154] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 22:19:48 (1789265988) [ 1039.601733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1041.228728] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 22:19:51 (1789265991) [ 1042.809736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1044.712319] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 22:19:54 (1789265994) [ 1046.314244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1048.405300] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 22:19:58 (1789265998) [ 1050.348732] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1052.384963] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 22:20:02 (1789266002) [ 1052.537168] LNet: 52703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.410539] Key type .llcrypt unregistered [ 1053.412149] Key type ._llcrypt unregistered [ 1061.101589] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1061.326766] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1061.337555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1061.351146] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1063.310815] Key type ._llcrypt registered [ 1063.315395] Key type .llcrypt registered [ 1063.591820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1063.604746] alg: No test for adler32 (adler32-zlib) [ 1064.649163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.859291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1064.918267] LNet: Added LNI 192.168.201.59@tcp [8/512/0/180] [ 1064.973440] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1064.978728] LNet: 53019:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1065.133771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.166767] LNet: 53038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.172143] LNetError: 53038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1065.178572] LNet: Removed LNI 10.1.2.3@tcp [ 1065.181405] LNetError: 53038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.192345] LNet: Removed LNI 192.168.201.59@tcp [ 1065.222710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.330679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1065.382422] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1065.528277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.589438] LNet: 53062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.598604] LNetError: 53062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.630364] LNet: Removed LNI 192.168.201.59@tcp [ 1065.658231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1065.719629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1065.748090] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1065.753047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1065.756495] LNet: 53070:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1065.852858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1065.874722] LNet: 53088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.880805] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1065.883912] LNet: Removed LNI 10.1.2.3@tcp [ 1065.885596] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.902667] LNet: Removed LNI 192.168.201.59@tcp [ 1065.944404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.028809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1066.052316] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1066.056803] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1066.060862] LNet: 53096:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1066.320865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.359659] LNet: 53122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.366463] LNetError: 53122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1066.376237] LNet: Removed LNI 10.1.2.3@tcp [ 1066.377704] LNetError: 53122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.422495] LNet: Removed LNI 192.168.201.59@tcp [ 1066.475490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.584492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1066.609088] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 1066.616707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1066.741266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.759412] LNet: 53150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.766833] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1066.772547] LNet: Removed LNI 10.1.2.3@tcp [ 1066.777651] LNetError: 53150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.803955] LNet: Removed LNI 192.168.201.59@tcp [ 1066.845971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.932311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1066.970873] LNet: Added LNI 192.168.201.59@tcp [8/256/0/182] [ 1066.985536] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1066.988036] LNet: 53158:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1067.066407] LNet: Removed LNI 10.1.2.3@tcp [ 1067.080312] LNet: Removed LNI 192.168.201.59@tcp [ 1067.873203] Key type .llcrypt unregistered [ 1067.875470] Key type ._llcrypt unregistered [ 1081.771821] Key type ._llcrypt registered [ 1081.774110] Key type .llcrypt registered [ 1082.851769] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 22:20:33 (1789266033) [ 1084.285818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1086.350796] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 22:20:36 (1789266036) [ 1086.655940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.671888] alg: No test for adler32 (adler32-zlib) [ 1087.641917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1087.698071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1087.721305] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1088.619430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.59 [ 1088.643681] LNet: 54420:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1088.650335] LNet: Removed LNI 192.168.201.59@tcp [ 1095.243573] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 22:20:45 (1789266045) [ 1095.642925] LNet: 54989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.653658] LNetError: 54989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.665758] LNet: Removed LNI 192.168.201.59@tcp [ 1096.292817] Key type .llcrypt unregistered [ 1096.296510] Key type ._llcrypt unregistered [ 1103.664031] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1104.031287] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1106.995327] Key type ._llcrypt registered [ 1106.998168] Key type .llcrypt registered [ 1107.203330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.217658] alg: No test for adler32 (adler32-zlib) [ 1108.617942] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1116.592307] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 22:21:06 (1789266066) [ 1116.800656] LNet: 55893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.808476] LNetError: 55893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1116.836381] LNet: Removed LNI 10.1.2.3@tcp [ 1117.542215] Key type .llcrypt unregistered [ 1117.543895] Key type ._llcrypt unregistered [ 1124.466991] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1124.715824] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1124.725137] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1124.739668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1127.545867] Key type ._llcrypt registered [ 1127.548893] Key type .llcrypt registered [ 1127.727806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1127.761952] alg: No test for adler32 (adler32-zlib) [ 1128.779600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1135.639626] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 22:21:25 (1789266085) [ 1137.090344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1138.571651] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 22:21:28 (1789266088) [ 1138.661201] LNet: 57071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.666944] LNetError: 57071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1138.681514] LNet: Removed LNI 10.1.2.3@tcp [ 1139.375644] Key type .llcrypt unregistered [ 1139.378379] Key type ._llcrypt unregistered [ 1147.031577] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1147.319869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1147.323412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1147.333975] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1150.109985] Key type ._llcrypt registered [ 1150.114957] Key type .llcrypt registered [ 1150.409868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.435887] alg: No test for adler32 (adler32-zlib) [ 1151.608799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1151.776738] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1162.482258] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 22:21:52 (1789266112) [ 1163.056883] LNet: 58308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.061782] LNetError: 58308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1163.072202] LNet: Removed LNI 10.1.2.3@tcp [ 1163.891576] Key type .llcrypt unregistered [ 1163.893221] Key type ._llcrypt unregistered [ 1170.574515] Key type ._llcrypt registered [ 1170.582994] Key type .llcrypt registered [ 1170.873849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1170.893657] alg: No test for adler32 (adler32-zlib) [ 1171.817290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1171.851898] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1180.158850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1181.007537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1181.049077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1181.095566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1181.123706] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1181.371771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.426925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.483370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.540745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.605093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.672941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.716807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.783470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.824881] LNet: 58743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.851699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.887233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.909379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.936957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.980859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.018652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.057225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.724929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1182.783367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1182.837861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1182.862457] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1183.045504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.071986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.106752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.137207] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1183.144734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.168373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.195382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.227365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.259812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.274792] LNet: 58852:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.313876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.362848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.389792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.427445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.456285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.485400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.530895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1184.076210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1184.103497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1184.149602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1184.167892] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1184.172993] LNet: Skipped 1 previous similar message [ 1184.403496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.435707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.457094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.511413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.544145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.579759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.603766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.627675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.642669] LNet: 58961:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.659923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.707384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.760259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.800706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.839067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.863737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.892340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.427947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1185.458430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1185.494442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1185.716402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1185.747341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1185.766342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1185.789900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1185.826437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1185.868483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1185.897930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1185.928475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1185.969893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.008459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.072765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.124104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.160412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.207648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.246219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.827910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1186.873682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1186.911907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1186.934491] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1186.939502] LNet: Skipped 1 previous similar message [ 1187.108600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1187.140479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1187.168394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1187.203763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1187.244226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1187.268723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1187.298985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1187.331514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1187.356791] LNet: 59180:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1187.366934] LNet: 59180:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1187.388744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1187.434523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1187.464721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1187.501210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1187.537472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1187.569389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1187.610072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1188.065296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1188.092241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1188.118519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1188.369236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.408458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.435332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.456638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.516577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.570771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.643992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.715653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.753321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.800802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.853560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.896426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.942432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.007887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.055900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1189.368408] LNet: 59330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.383287] LNetError: 59330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1189.408251] LNet: Removed LNI 192.168.201.59@tcp [ 1189.913589] Key type .llcrypt unregistered [ 1189.919290] Key type ._llcrypt unregistered [ 1209.652289] Key type ._llcrypt registered [ 1209.653659] Key type .llcrypt registered [ 1211.538318] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 22:22:40 (1789266160) [ 1212.501476] Key type .llcrypt unregistered [ 1212.503228] Key type ._llcrypt unregistered [ 1221.174930] Key type ._llcrypt registered [ 1221.177815] Key type .llcrypt registered [ 1221.538499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1221.565963] alg: No test for adler32 (adler32-zlib) [ 1222.613218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1222.648387] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1231.868107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1246.950861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.041360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1247.090257] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1249.437971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1249.534671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 1249.610776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1249.713247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 1249.817213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 1 [ 1249.916592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 1 [ 1250.015977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 1 [ 1250.114156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 1 [ 1250.248573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1250.304119] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1250.315016] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1250.673844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1250.755112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1250.806502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1250.882517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.981190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1251.026560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1251.061746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1251.107821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1251.145837] LNet: 60798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1251.177884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1251.259950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1251.333063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1251.363683] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1251.374800] LNet: Skipped 3 previous similar messages [ 1251.437319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1251.512133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1251.575176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1251.652407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1252.697664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1252.774242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 1252.847282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1252.922983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 1253.020742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 1 [ 1253.105613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 1 [ 1253.182255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 1 [ 1253.254342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 1 [ 1253.348505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1253.413069] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1254.003457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1254.067918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1254.157930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1254.228728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1254.299146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1254.370185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1254.459287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1254.566552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1254.617853] LNet: 60920:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1254.672836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1254.754892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1254.872533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1254.963348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1255.037774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1255.117553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.172576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1256.259589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1256.328141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 1256.409205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1256.501426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 1256.588382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 1 [ 1256.687760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 1 [ 1256.780121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 1 [ 1256.835443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 1 [ 1256.872686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1256.896988] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1256.906287] LNet: Skipped 2 previous similar messages [ 1257.334217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1257.416826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1257.487780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1257.565697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1257.677825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1257.747528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1257.786514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1257.838534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1257.858233] LNet: 61041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1257.889171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1257.934124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1257.971547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1258.014816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.059519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1258.152457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1258.208476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1258.807364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1258.841830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 1258.878742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1258.938786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 1258.981217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 1 [ 1259.020747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 1 [ 1259.061516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 1 [ 1259.101172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 1 [ 1259.158846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1259.605698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.630402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.653717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.677567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.704528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.728471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.755993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.781897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.812409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.836392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1259.868577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1259.898855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1259.971672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.097652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.133464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1260.864658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1260.896874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 1260.933443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1260.962701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 1261.000307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 1 [ 1261.038067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 1 [ 1261.075440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 1 [ 1261.124202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 1 [ 1261.174496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1261.197668] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1261.203965] LNet: Skipped 5 previous similar messages [ 1261.518982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.610852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.691927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.737977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.785662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.865690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.933235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.994192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.027500] LNet: 61284:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.039287] LNet: 61284:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1262.074963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.126636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.167129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.213433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.245641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.307461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.343168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1262.885630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1262.932922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 1262.969858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1263.005801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 1263.041589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_error -r 1 [ 1263.077687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_error -r 1 [ 1263.112698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_error -r 1 [ 1263.141092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_error -r 1 [ 1263.176783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1263.611818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.660834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1263.757228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.833902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.894380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.967496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1264.042691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1264.090203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1264.131115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1264.181757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1264.262968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.331455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1264.403944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1264.473765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1264.560236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1273.541606] LNet: 61482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.559522] LNetError: 61482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.591452] LNet: Removed LNI 192.168.201.59@tcp [ 1274.595777] LNetError: 61482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1274.625180] LNet: Removed LNI 192.168.201.59@tcp1 [ 1275.352464] Key type .llcrypt unregistered [ 1275.357406] Key type ._llcrypt unregistered [ 1296.985558] Key type ._llcrypt registered [ 1296.989458] Key type .llcrypt registered [ 1298.794819] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 22:24:08 (1789266248) [ 1299.722401] Key type .llcrypt unregistered [ 1299.726574] Key type ._llcrypt unregistered [ 1307.466078] Key type ._llcrypt registered [ 1307.477396] Key type .llcrypt registered [ 1307.754723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1307.783344] alg: No test for adler32 (adler32-zlib) [ 1308.767907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1308.804118] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1317.089685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1317.948634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1317.982627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1318.019899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1318.039430] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1318.285770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1318.325554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1318.356399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.385826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.412064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.441467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.477581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.530551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.559131] LNet: 62727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.589453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.624987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.660713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.689116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.723564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.765340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.813544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1319.430587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1319.458027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1319.486411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1319.503143] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1319.702162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1319.740484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1319.773493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1319.804522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1319.838389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1319.869406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1319.905100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1319.939696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1319.968034] LNet: 62836:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1319.984966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1320.021292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1320.055788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1320.088412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1320.115361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1320.148304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1320.176788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1320.697472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1320.732259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1320.762615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1320.779448] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1321.028598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1321.058403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1321.084751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1321.112273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1321.139634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1321.170780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1321.202372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1321.232628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1321.248798] LNet: 62945:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1321.265985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1321.301961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1321.329756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1321.369191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1321.395623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1321.430386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1321.484573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1321.898630] LNet: 62986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.904113] LNetError: 62986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1321.943584] LNet: Removed LNI 192.168.201.59@tcp [ 1322.400885] Key type .llcrypt unregistered [ 1322.403414] Key type ._llcrypt unregistered [ 1340.312544] Key type ._llcrypt registered [ 1340.315524] Key type .llcrypt registered [ 1341.645053] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 22:24:51 (1789266291) [ 1342.600421] Key type .llcrypt unregistered [ 1342.604150] Key type ._llcrypt unregistered [ 1350.480947] Key type ._llcrypt registered [ 1350.484096] Key type .llcrypt registered [ 1350.650729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1350.670528] alg: No test for adler32 (adler32-zlib) [ 1351.730148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1351.764628] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1360.575963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1371.897304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1371.967543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1371.997451] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1373.948627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1373.997601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1374.044231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1374.113581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1374.184513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 1 [ 1374.244426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 1 [ 1374.294472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 1 [ 1374.339874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 1 [ 1374.367442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1374.381663] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1374.386654] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1374.572901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1374.601865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1374.637595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1374.674395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1374.710336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1374.751663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1374.797108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1374.858632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1374.892941] LNet: 64448:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1374.922627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1374.958093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1374.999741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1375.041636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1375.103727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1375.142239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1375.180368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1375.226602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.076910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1376.155529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1376.204408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1376.239222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1376.280450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_error -r 1 [ 1376.324963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_error -r 1 [ 1376.371435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_error -r 1 [ 1376.429699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_error -r 1 [ 1376.474764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1376.501682] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1376.507731] LNet: Skipped 2 previous similar messages [ 1376.768335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1376.834155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1376.879538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1376.932562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1377.000071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1377.042387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1377.083255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1377.116664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1377.136666] LNet: 64572:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1377.170358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1377.213502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1377.243493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1377.286411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1377.314853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1377.356244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1377.415933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1377.459050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1378.070312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1378.125437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1378.179437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1378.234949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1378.293746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_timeout -r 1 [ 1378.323122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_timeout -r 1 [ 1378.404109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_timeout -r 1 [ 1378.468086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_timeout -r 1 [ 1378.575275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.159@tcp [ 1378.620498] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1378.966902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1378.999641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1379.076327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1379.114138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1379.160522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1379.213720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1379.263437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1379.299773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1379.314834] LNet: 64695:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1379.346185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1379.391894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1379.435818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1379.474634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.504619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.542354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.573572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1379.619798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1384.535929] LNet: 64773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.546365] LNetError: 64773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.555822] LNet: Removed LNI 192.168.201.59@tcp [ 1384.570566] LNet: Removed LNI 192.168.201.59@tcp1 [ 1385.197853] Key type .llcrypt unregistered [ 1385.201519] Key type ._llcrypt unregistered [ 1403.269720] Key type ._llcrypt registered [ 1403.272568] Key type .llcrypt registered [ 1404.888213] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 22:25:54 (1789266354) [ 1406.272947] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1408.026445] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 22:25:58 (1789266358) [ 1408.855860] Key type .llcrypt unregistered [ 1408.860918] Key type ._llcrypt unregistered [ 1415.112244] Key type ._llcrypt registered [ 1415.113717] Key type .llcrypt registered [ 1415.433354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1415.461242] alg: No test for adler32 (adler32-zlib) [ 1416.418220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1416.447233] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1424.246581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1424.944346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1424.977939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1425.008950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1425.024505] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1425.219688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1425.243937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1425.267182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1425.287930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1425.311326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1425.333578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1425.360751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1425.383706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1425.399533] LNet: 66312:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1425.416919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1425.439739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1425.460827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1425.485332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1425.507837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1425.536725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1425.713370] LNet: 66351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.720574] LNetError: 66351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.804060] LNet: Removed LNI 192.168.201.59@tcp [ 1427.252996] Key type .llcrypt unregistered [ 1427.256141] Key type ._llcrypt unregistered [ 1445.726802] Key type ._llcrypt registered [ 1445.729365] Key type .llcrypt registered [ 1445.967353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1446.010128] alg: No test for adler32 (adler32-zlib) [ 1446.936542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1446.959439] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1454.322561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1463.125868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.183905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1463.217172] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1465.160630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1465.223105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1465.291660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1465.363305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1465.440042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e network_timeout -r 1 [ 1465.509971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e network_timeout -r 1 [ 1465.588273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e network_timeout -r 1 [ 1465.656977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e network_timeout -r 1 [ 1465.731715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1465.759394] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1466.131894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1466.176499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1466.237993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1466.276913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1466.308445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1466.358130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1466.413150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1466.468863] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1466.470195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1466.471974] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1466.479984] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1466.527411] LNet: 67226:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.556523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1466.613443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1466.652937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1466.695281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1466.718315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1466.753636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1471.619315] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1471.622215] LNet: Skipped 1 previous similar message [ 1471.926795] LNet: 67300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.932795] LNetError: 67300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.945507] LNet: Removed LNI 192.168.201.59@tcp [ 1471.956554] LNet: Removed LNI 192.168.201.59@tcp1 [ 1472.388661] Key type .llcrypt unregistered [ 1472.392032] Key type ._llcrypt unregistered [ 1488.662181] Key type ._llcrypt registered [ 1488.663959] Key type .llcrypt registered [ 1490.148590] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 22:27:20 (1789266440) [ 1490.724434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1490.743795] alg: No test for adler32 (adler32-zlib) [ 1491.749298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.827882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.858270] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1492.944774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1492.970715] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1494.253273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1494.315382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1494.364932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1494.397870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1494.436579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1494.476392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1494.519617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1494.557145] LNetError: 68398:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1494.879395] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1494.880498] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1496.928259] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1498.072720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1501.029350] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1501.690218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1501.744482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1501.806915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.829186] LNet: 68505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.833948] LNetError: 68505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1501.845774] LNet: Removed LNI 192.168.201.59@tcp [ 1501.856775] LNet: Removed LNI 192.168.201.59@tcp1 [ 1501.877684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.924494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1501.952897] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1502.500947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1502.515829] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1503.374969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1503.455486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1503.535700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1503.586469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1503.620314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1503.658846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1503.690185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1503.721270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1503.739762] LNetError: 68778:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1503.968056] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1506.212318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1510.112366] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1510.124316] LNet: Skipped 1 previous similar message [ 1515.309571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1515.359212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1515.403555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1522.302697] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 22:27:52 (1789266472) [ 1522.496760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.523378] LNet: 69492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.540064] LNetError: 69492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.550375] LNetError: 69492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1522.572859] LNet: Removed LNI 192.168.201.59@tcp [ 1522.608674] LNet: Removed LNI 192.168.201.59@tcp1 [ 1522.647642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.701053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.724261] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1523.373776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1523.388307] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1524.235628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1524.299965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1524.327621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1524.363930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1524.400674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1524.431799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1524.486600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1524.595549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1524.725956] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.59@tcp [ 1525.802949] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1525.816766] LNet: Skipped 1 previous similar message [ 1539.593857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.886609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1539.939878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1539.987553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 500 [ 1540.128224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.968852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1547.082856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.102391] LNet: 69980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.112191] LNetError: 69980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.115282] LNetError: 69980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1547.123465] LNet: Removed LNI 192.168.201.59@tcp [ 1547.141803] LNet: Removed LNI 192.168.201.59@tcp1 [ 1547.188774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.242728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.261214] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1547.772751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1547.786405] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1548.513870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1548.559634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1548.628874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1548.663412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1548.711704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1548.773817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1548.815515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1548.853558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.59@tcp --health 0 [ 1549.282967] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1549.296910] LNet: Skipped 3 previous similar messages [ 1550.234582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1559.820358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1559.868983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1559.930300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1567.389473] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 22:28:37 (1789266517) [ 1567.773808] LNet: 71014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.778765] LNetError: 71014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.783427] LNetError: 71014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1567.787448] LNet: Removed LNI 192.168.201.59@tcp [ 1567.804422] LNet: Removed LNI 192.168.201.59@tcp1 [ 1568.246235] Key type .llcrypt unregistered [ 1568.248440] Key type ._llcrypt unregistered [ 1575.738911] Key type ._llcrypt registered [ 1575.744259] Key type .llcrypt registered [ 1576.004735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1576.016859] alg: No test for adler32 (adler32-zlib) [ 1576.963585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1577.004224] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1586.313985] Lustre: DEBUG MARKER: Initial discovery [ 1586.398486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1589.026738] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1589.238812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1589.298855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1589.353272] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1589.397100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1589.447703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -l 3 [ 1589.499659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.159@tcp -r 1 -m GET -e local_error [ 1589.550833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.159@tcp -r 1 -m PUT -l 6 [ 1590.304561] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1591.046641] Lustre: DEBUG MARKER: Force oleg159-client.virtnet to discover 192.168.201.159@tcp (in background) [ 1591.301984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.159@tcp [ 1592.896468] Lustre: DEBUG MARKER: Force oleg159-server to discover 192.168.201.59@tcp [ 1594.407074] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1595.210829] Lustre: DEBUG MARKER: Wait for 71776 [ 1596.531734] Lustre: DEBUG MARKER: Finished wait on 71776 [ 1596.851931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1596.893874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1596.894993] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1597.188252] LNet: 72225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.196245] LNetError: 72225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.219193] LNet: Removed LNI 192.168.201.59@tcp [ 1597.567641] Key type .llcrypt unregistered [ 1597.579592] Key type ._llcrypt unregistered [ 1613.711392] Key type ._llcrypt registered [ 1613.713760] Key type .llcrypt registered [ 1615.046395] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 22:29:25 (1789266565) [ 1615.590359] Key type .llcrypt unregistered [ 1615.592036] Key type ._llcrypt unregistered [ 1622.211675] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1622.497817] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1622.519875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1622.529181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1625.262213] Key type ._llcrypt registered [ 1625.264527] Key type .llcrypt registered [ 1625.568774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1625.630657] alg: No test for adler32 (adler32-zlib) [ 1626.714662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.805807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.877084] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1628.210864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1628.230883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1640.937464] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 22:29:51 (1789266591) [ 1641.085210] LNet: 74293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.096896] LNetError: 74293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1641.101046] LNet: Removed LNI 10.1.2.3@tcp [ 1641.130919] LNet: Removed LNI 192.168.201.59@tcp [ 1641.829503] Key type .llcrypt unregistered [ 1641.832457] Key type ._llcrypt unregistered [ 1648.739345] Key type ._llcrypt registered [ 1648.741018] Key type .llcrypt registered [ 1649.136616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1649.164029] alg: No test for adler32 (adler32-zlib) [ 1650.089711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.159479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.198298] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1650.915602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1650.940767] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1652.134466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.59@tcp [ 1653.220685] LNetError: Unexpected error -22 connecting to 192.168.201.59@tcp at host 192.168.201.59:988 [ 1659.028485] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 22:30:08 (1789266608) [ 1659.173484] LNet: 75422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.184682] LNetError: 75422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1659.197760] LNet: Removed LNI 10.1.2.3@tcp [ 1659.260475] LNet: Removed LNI 192.168.201.59@tcp [ 1660.209081] Key type .llcrypt unregistered [ 1660.210586] Key type ._llcrypt unregistered [ 1667.613694] Key type ._llcrypt registered [ 1667.617757] Key type .llcrypt registered [ 1667.947369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1667.982932] alg: No test for adler32 (adler32-zlib) [ 1668.989785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.159534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1669.231617] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1670.765818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1670.826598] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 1672.475945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.59@tcp1 --nid 192.168.201.59@tcp2 [ 1678.701110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp2 [ 1678.717912] LNetError: 76261:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.59@tcp1 to 192.168.201.59@tcp2 - Route table may be misconfigured [ 1678.746492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp2 192.168.201.59@tcp1 [ 1686.617266] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 22:30:36 (1789266636) [ 1686.774461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.800304] LNet: 76832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.821449] LNetError: 76832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.830043] LNet: Removed LNI 192.168.201.59@tcp1 [ 1687.848694] LNet: Removed LNI 192.168.201.59@tcp2 [ 1687.897747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.972347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1688.015588] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1688.783449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.801444] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1689.839693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 1689.917078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.59@tcp [ 1689.958904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp1 192.168.201.59@tcp1 [ 1690.045803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp -e network_timeout [ 1690.112740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -e network_timeout [ 1690.172191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -e network_timeout [ 1690.283323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -e network_timeout [ 1690.356243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1690.386894] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1690.406179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1690.548277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1690.668492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1691.040212] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1697.634785] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 22:30:47 (1789266647) [ 1697.758077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.787695] LNet: 77691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.813863] LNetError: 77691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.823390] LNetError: 77691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1697.842084] LNet: Removed LNI 192.168.201.59@tcp [ 1698.924339] LNet: Removed LNI 192.168.201.59@tcp1 [ 1698.976860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.060729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1699.323419] LNet: 77733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.784068] Key type .llcrypt unregistered [ 1699.786432] Key type ._llcrypt unregistered [ 1710.700892] Key type ._llcrypt registered [ 1710.702720] Key type .llcrypt registered [ 1712.433652] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 22:31:02 (1789266662) [ 1713.025686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1713.046183] alg: No test for adler32 (adler32-zlib) [ 1713.993704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.464616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1717.214921] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 22:31:07 (1789266667) [ 1717.458504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.472774] LNet: 78828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.496636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.529946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.544811] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1718.431807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1718.462140] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1719.330717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1719.362595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp1 [ 1719.398794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp1 [ 1725.399913] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 22:31:15 (1789266675) [ 1727.179321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg159-server [ 1729.180929] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 22:31:19 (1789266679) [ 1730.860574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg159-server [ 1732.394042] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 22:31:22 (1789266682) [ 1734.060822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg159-server [ 1735.826457] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 22:31:25 (1789266685) [ 1737.967301] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg159-server [ 1739.604060] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 22:31:29 (1789266689) [ 1741.690224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg159-server [ 1744.013158] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 22:31:33 (1789266693) [ 1746.625224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg159-server [ 1748.881169] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 22:31:38 (1789266698) [ 1751.603851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg159-server [ 1754.045809] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 22:31:43 (1789266703) [ 1755.658950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg159-server [ 1757.377502] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 22:31:47 (1789266707) [ 1759.039980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg159-server [ 1760.480874] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 22:31:50 (1789266710) [ 1760.746211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.789075] LNet: 82706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.810770] LNetError: 82706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.823647] LNet: Removed LNI 192.168.201.59@tcp [ 1761.825567] LNetError: 82706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.862945] LNet: Removed LNI 192.168.201.59@tcp1 [ 1761.903544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.969081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.008164] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1762.607265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1762.683358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1762.835305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.852744] LNet: 82853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.858221] LNetError: 82853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.868928] LNet: Removed LNI 192.168.201.59@tcp [ 1762.888960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.936216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1762.967978] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1763.801985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1763.993961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.111216] LNet: Removed LNI 192.168.201.59@tcp [ 1765.193757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.376942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.435284] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1766.364779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1766.487981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1766.812161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1766.858306] LNet: 83146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.863622] LNet: 83146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1766.871720] LNetError: 83146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.879564] LNetError: 83146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1766.895459] LNet: Removed LNI 192.168.201.59@tcp [ 1766.924674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.009590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1767.041405] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1767.836324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1768.122273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.198327] LNet: Removed LNI 192.168.201.59@tcp [ 1769.238280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.329978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.365804] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1770.190840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1770.322961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1770.617225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1771.694847] LNet: Removed LNI 192.168.201.59@tcp [ 1771.768524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1771.876940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1771.931528] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1773.261175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1773.589877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.645546] LNet: 83584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.649371] LNet: 83584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1773.660120] LNetError: 83584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.681552] LNetError: 83584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1773.734870] LNet: Removed LNI 192.168.201.59@tcp [ 1773.785983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.878765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.909120] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1774.869044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1774.931859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1775.193290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.295530] LNet: Removed LNI 192.168.201.59@tcp [ 1776.342303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.438384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1776.479927] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1777.604380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1777.977623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.048938] LNet: Removed LNI 192.168.201.59@tcp [ 1779.068785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.122540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.160414] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1779.904279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1780.044686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1780.388593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.480848] LNet: Removed LNI 192.168.201.59@tcp [ 1781.548828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.651869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1781.691344] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1783.012328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1783.466689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.487100] LNet: 84169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.489982] LNet: 84169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1783.494987] LNetError: 84169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.504467] LNetError: 84169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1783.517512] LNet: Removed LNI 192.168.201.59@tcp [ 1783.547461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.620644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.658600] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1784.799305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1784.923578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1785.215875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1786.278251] LNet: Removed LNI 192.168.201.59@tcp [ 1786.319769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.407757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1786.437297] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1788.070098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 1788.581118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1789.687615] LNet: Removed LNI 192.168.201.59@tcp [ 1789.747606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1789.939986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1790.011345] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1791.574753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1791.755638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1791.830618] LNet: Removed LNI 192.168.201.59@tcp [ 1791.903777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1791.996622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.054987] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1793.337263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1793.447833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.528900] LNet: Removed LNI 192.168.201.59@tcp [ 1793.572432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.672358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1793.800509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.861683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.964306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1794.099861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.143790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.217351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1794.322290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.360763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.415755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1794.461706] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1795.400453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.428684] LNet: Removed LNI 192.168.201.59@tcp [ 1795.445826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.545753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1804.234511] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 22:32:34 (1789266754) [ 1804.370534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.409744] LNet: 85475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.419981] LNet: 85475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1804.457272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.544738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1804.586728] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1805.913176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1805.936440] LNetError: 85621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.942319] LNetError: 85621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1805.956515] LNet: Removed LNI 192.168.201.59@tcp231 [ 1805.972213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.063971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1806.125242] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1806.139359] LNet: 85629:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1806.848828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1806.900510] LNet: Removed LNI 192.168.201.59@tcp231 [ 1806.926641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1806.972193] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1808.613985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.717213] LNet: Removed LNI 192.168.201.59@tcp231 [ 1808.742736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.900829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1808.939838] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1808.952495] LNet: 85782:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1809.871303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1809.928986] LNet: Removed LNI 192.168.201.59@tcp231 [ 1809.968367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1810.029151] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/0] [ 1811.377430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1811.413688] LNet: Removed LNI 192.168.201.59@tcp231 [ 1811.435291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1811.544272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1811.588490] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1811.592190] LNet: 85933:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1811.835547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1811.865991] LNet: Removed LNI 192.168.201.59@tcp231 [ 1811.880485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1811.896046] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/60] [ 1813.168118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.195428] LNet: Removed LNI 192.168.201.59@tcp231 [ 1813.236115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.333631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1813.393981] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1813.844544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1813.896288] LNet: Removed LNI 192.168.201.59@tcp231 [ 1813.921130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1813.955142] LNet: Added LNI 192.168.201.59@tcp231 [8/256/0/180] [ 1822.984430] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 22:32:53 (1789266773) [ 1823.154971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.191782] LNet: Removed LNI 192.168.201.59@tcp231 [ 1823.210596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.276053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1823.301008] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1824.098796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1824.150380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1824.179409] LNet: Removed LNI 192.168.201.59@tcp232 [ 1824.192429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1824.211984] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1824.970813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1825.015414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1825.084733] LNet: Removed LNI 192.168.201.59@tcp232 [ 1825.110445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1825.166757] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1826.037039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1826.112730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1826.151660] LNet: Removed LNI 192.168.201.59@tcp232 [ 1826.167667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1826.204078] LNet: Added LNI 192.168.201.59@tcp232 [8/256/0/180] [ 1826.949906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1826.978591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1827.004940] LNet: Removed LNI 192.168.201.59@tcp232 [ 1827.011827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1832.959561] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 22:33:03 (1789266783) [ 1833.974875] Key type .llcrypt unregistered [ 1833.976638] Key type ._llcrypt unregistered [ 1839.922917] Key type ._llcrypt registered [ 1839.929589] Key type .llcrypt registered [ 1840.169806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1840.181362] alg: No test for adler32 (adler32-zlib) [ 1841.162244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1841.195800] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1849.838389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1859.670559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.771582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1859.801324] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1861.754779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1861.786697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1861.823845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1861.853277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1861.892444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_interrupt -r 2 [ 1861.980356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_interrupt -r 2 [ 1862.030615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_interrupt -r 2 [ 1862.096732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_interrupt -r 2 [ 1862.150518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1862.203928] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1862.220844] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1862.560917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1862.602244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1862.632713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1862.661795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1862.696196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1862.735552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1862.770832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1862.803815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1862.820686] LNet: 88584:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1862.838614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1862.861830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1862.890196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1862.926170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1862.954751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1862.983608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1863.670681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1863.740302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1863.800030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1863.851753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1863.912120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_dropped -r 2 [ 1863.972256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_dropped -r 2 [ 1864.042284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_dropped -r 2 [ 1864.134166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_dropped -r 2 [ 1864.187240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1864.220148] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1864.563261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1864.616626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1864.657232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1864.695349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1864.729685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1864.763270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1864.812738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1864.858386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1864.880571] LNet: 88704:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1864.907595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1864.931652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1864.955328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1864.982800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1865.024543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1865.071412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1865.721744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1865.772403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1865.816909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1865.918916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1866.002438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_aborted -r 2 [ 1866.076427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_aborted -r 2 [ 1866.137776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_aborted -r 2 [ 1866.207727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_aborted -r 2 [ 1866.290770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1866.343239] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1866.691813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1866.752981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1866.805816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1866.850304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1866.901345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1866.934070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1866.979714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1867.026307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1867.043771] LNet: 88824:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1867.078657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1867.106408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1867.131948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1867.165207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1867.188575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1867.209988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1867.848907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1867.884688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1867.920837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1867.952718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1867.990816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_no_route -r 2 [ 1868.043670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_no_route -r 2 [ 1868.086366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_no_route -r 2 [ 1868.115435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_no_route -r 2 [ 1868.158945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1868.387717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1868.414300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1868.442901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1868.472980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1868.502918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1868.526969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1868.559669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1868.592952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1868.624617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1868.652256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1868.682726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1868.708221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1868.746387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1868.787411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1869.258251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1869.295909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1869.324280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1869.353969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1869.383682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e local_timeout -r 2 [ 1869.411266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e local_timeout -r 2 [ 1869.440355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e local_timeout -r 2 [ 1869.473192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e local_timeout -r 2 [ 1869.503509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1869.531133] LNet: There was an unexpected network error while writing to 192.168.201.159: rc = -22 [ 1869.533759] LNet: Skipped 1 previous similar message [ 1869.758474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1869.780491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1869.805327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1869.828115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1869.854703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1869.878581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1869.908937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1869.937478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1869.955224] LNet: 89065:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1869.959519] LNet: 89065:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1869.977927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1870.005520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1870.028857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1870.053921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1870.115559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1870.155053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1870.708354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 1870.740785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 1870.771192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 1870.794955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 1870.815551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp -m GET -e remote_dropped -r 2 [ 1870.838231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.59@tcp1 -d 192.168.201.159@tcp1 -m GET -e remote_dropped -r 2 [ 1870.868036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp -m GET -e remote_dropped -r 2 [ 1870.914932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.59@tcp1 -d 192.168.201.59@tcp1 -m GET -e remote_dropped -r 2 [ 1870.965786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.59@tcp 192.168.201.159@tcp [ 1870.989970] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 1871.237530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1871.273704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1871.322759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1871.350167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1871.368407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1871.403162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1871.431568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1871.459243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1871.485633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1871.517617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1871.549658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1871.571851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1871.598688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1871.626232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1878.267509] LNet: 89260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.275635] LNetError: 89260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.328987] LNet: Removed LNI 192.168.201.59@tcp [ 1879.331368] LNetError: 89260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1879.359824] LNet: Removed LNI 192.168.201.59@tcp1 [ 1879.952756] Key type .llcrypt unregistered [ 1879.954564] Key type ._llcrypt unregistered [ 1896.716658] Key type ._llcrypt registered [ 1896.718630] Key type .llcrypt registered [ 1898.280555] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 22:34:08 (1789266848) [ 1898.601719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1898.622159] alg: No test for adler32 (adler32-zlib) [ 1899.566136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.596491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1899.618660] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1900.504135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1900.549627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.59@tcp [ 1900.585346] LNetError: 90237:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1900.643455] LNet: 1 local NIs in recovery (showing 1): 192.168.201.59@tcp [ 1900.715152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.59@tcp [ 1900.820443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1907.630654] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 22:34:17 (1789266857) [ 1907.772647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.810144] LNet: 90824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1907.817564] LNetError: 90824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.840755] LNet: Removed LNI 192.168.201.59@tcp [ 1907.908442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.970604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.994937] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1908.855528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1908.890571] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 1910.275188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.59@tcp [ 1910.306974] LNetError: 91093:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.59@tcp to recovery [ 1911.009232] LNet: There was an unexpected network error while writing to 192.168.201.59: rc = -22 [ 1911.593598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.59@tcp [ 1921.227598] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 22:34:31 (1789266871) [ 1923.056768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg159-server [ 1924.394503] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 22:34:34 (1789266874) [ 1926.286728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg159-server [ 1928.067990] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 22:34:38 (1789266878) [ 1929.795029] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg159-server [ 1931.798407] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 22:34:41 (1789266881) [ 1932.140534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.161419] LNet: 92723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.177518] LNetError: 92723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.201876] LNet: Removed LNI 192.168.201.59@tcp [ 1932.227782] LNet: Removed LNI 192.168.201.59@tcp1 [ 1932.241109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.289782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.310324] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1933.113808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1933.191508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1933.250399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1933.389891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1933.439969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1933.522398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1940.771155] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 22:34:50 (1789266890) [ 1940.914660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.995665] LNet: 93447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.003610] LNetError: 93447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.008914] LNetError: 93447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1941.053430] LNet: Removed LNI 192.168.201.59@tcp [ 1941.150719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.261520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.305866] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1948.401940] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 22:34:58 (1789266898) [ 1948.627201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.643655] LNet: 94158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.649236] LNetError: 94158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.664511] LNet: Removed LNI 192.168.201.59@tcp [ 1948.690923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.742983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1948.788463] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1955.253527] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 22:35:05 (1789266905) [ 1956.916267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1959.440827] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 22:35:09 (1789266909) [ 1960.055410] LNet: 95192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1960.062864] LNetError: 95192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1960.083891] LNet: Removed LNI 192.168.201.59@tcp [ 1960.784704] Key type .llcrypt unregistered [ 1960.793233] Key type ._llcrypt unregistered [ 1967.675158] Key type ._llcrypt registered [ 1967.676390] Key type .llcrypt registered [ 1967.926947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1967.942600] alg: No test for adler32 (adler32-zlib) [ 1968.929498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1968.987188] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 1978.398228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 1984.861965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.159@tcp [ 1985.235398] LNet: 95607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.240880] LNetError: 95607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.252695] LNet: Removed LNI 192.168.201.59@tcp [ 1985.731744] Key type .llcrypt unregistered [ 1985.734405] Key type ._llcrypt unregistered [ 1999.453354] Key type ._llcrypt registered [ 1999.455462] Key type .llcrypt registered [ 2000.957500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2002.651729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2004.451550] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 22:35:54 (1789266954) [ 2006.647347] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2008.614231] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 22:35:58 (1789266958) [ 2011.551224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg159-server [ 2014.006423] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 22:36:03 (1789266963) [ 2016.418607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg159-server [ 2018.151832] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 22:36:08 (1789266968) [ 2020.027555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.080382] alg: No test for adler32 (adler32-zlib) [ 2021.336083] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2021.586233] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2023.359755] Key type lgssc registered [ 2025.078668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2046.681917] Key type lgssc unregistered [ 2046.925077] LNet: 98779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.937399] LNetError: 98779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.952146] LNet: Removed LNI 192.168.201.59@tcp [ 2047.660460] Key type .llcrypt unregistered [ 2047.666387] Key type ._llcrypt unregistered [ 2062.390617] Key type ._llcrypt registered [ 2062.391779] Key type .llcrypt registered [ 2063.636899] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 22:36:53 (1789267013) [ 2064.130949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.154610] alg: No test for adler32 (adler32-zlib) [ 2065.115266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.203475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2065.251781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2065.302681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2065.344964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2071.930450] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 22:37:02 (1789267022) [ 2072.158898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.182258] LNet: 100192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.212977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.253207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2072.268581] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2073.078369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.097145] LNet: 100336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.103583] LNetError: 100336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.124029] LNet: Removed LNI 192.168.201.59@tcp [ 2073.148026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.211159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2073.239801] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2073.856470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.889507] LNetError: 100364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.902562] LNet: Removed LNI 192.168.201.59@tcp [ 2073.922279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2073.976268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2073.994309] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2081.019527] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 22:37:11 (1789267031) [ 2081.446944] LNet: 100954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2081.458325] LNet: 100954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2081.480717] LNetError: 100954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2081.501052] LNet: Removed LNI 192.168.201.59@tcp [ 2082.373652] Key type .llcrypt unregistered [ 2082.375954] Key type ._llcrypt unregistered [ 2089.196698] Key type ._llcrypt registered [ 2089.197758] Key type .llcrypt registered [ 2089.543292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2089.570956] alg: No test for adler32 (adler32-zlib) [ 2090.486617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.579173] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2090.584556] LNet: *** cfs_fail_loc=e003, val=100*** [ 2091.514936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2091.543136] LNet: *** cfs_fail_loc=e003, val=100*** [ 2091.546726] LNet: Skipped 2 previous similar messages [ 2091.562100] Ping reply received with mapping: [ 2091.564693] found metadata reply NI[192.168.201.59@tcp] [ 2091.571721] 3b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2091.582692] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2091.587698] 5a 5a 5a 5a 5a 5a 5a 5a [ 2091.808805] LNet: 101349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.817715] LNetError: 101349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.840658] LNet: Removed LNI 192.168.201.59@tcp [ 2093.661617] Key type .llcrypt unregistered [ 2093.664047] Key type ._llcrypt unregistered [ 2105.772700] Key type ._llcrypt registered [ 2105.774354] Key type .llcrypt registered [ 2107.145525] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 22:37:37 (1789267057) [ 2107.674414] Key type .llcrypt unregistered [ 2107.677178] Key type ._llcrypt unregistered [ 2114.431307] Key type ._llcrypt registered [ 2114.434526] Key type .llcrypt registered [ 2114.637301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2114.652471] alg: No test for adler32 (adler32-zlib) [ 2115.736054] LNetError: 102385:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2116.149824] Key type .llcrypt unregistered [ 2116.152816] Key type ._llcrypt unregistered [ 2121.709132] Key type ._llcrypt registered [ 2121.711098] Key type .llcrypt registered [ 2122.958508] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 22:37:53 (1789267073) [ 2123.563299] Key type .llcrypt unregistered [ 2123.565685] Key type ._llcrypt unregistered [ 2130.566377] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2130.579539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2132.152996] Key type ._llcrypt registered [ 2132.160074] Key type .llcrypt registered [ 2132.310378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2132.322821] alg: No test for adler32 (adler32-zlib) [ 2133.390650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.471663] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2143.584614] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 22:38:13 (1789267093) [ 2143.864551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.883501] LNet: 104150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.888450] LNetError: 104150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2143.927910] LNet: Removed LNI 10.1.2.3@tcp [ 2143.973678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.680889] Key type .llcrypt unregistered [ 2144.682092] Key type ._llcrypt unregistered [ 2152.270257] Key type ._llcrypt registered [ 2152.278470] Key type .llcrypt registered [ 2152.529931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2152.574342] alg: No test for adler32 (adler32-zlib) [ 2153.749562] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2153.816474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2154.000205] LNet: 104436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.009619] LNetError: 104436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.023767] LNet: Removed LNI 192.168.201.59@tcp [ 2155.294581] Key type .llcrypt unregistered [ 2155.297472] Key type ._llcrypt unregistered [ 2170.162946] Key type ._llcrypt registered [ 2170.164221] Key type .llcrypt registered [ 2171.700760] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 22:38:41 (1789267121) [ 2172.742330] Key type .llcrypt unregistered [ 2172.744699] Key type ._llcrypt unregistered [ 2180.868162] Key type ._llcrypt registered [ 2180.869766] Key type .llcrypt registered [ 2181.224863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2181.269909] alg: No test for adler32 (adler32-zlib) [ 2189.317799] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 22:38:59 (1789267139) [ 2189.775877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.820723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.898899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.947463] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2190.541877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2190.564430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2190.587845] LNet: Removed LNI 192.168.201.59@tcp [ 2190.876982] LNet: 106175:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2191.287640] Key type .llcrypt unregistered [ 2191.290550] Key type ._llcrypt unregistered [ 2202.374100] Key type ._llcrypt registered [ 2202.375466] Key type .llcrypt registered [ 2203.668395] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 22:39:13 (1789267153) [ 2204.240714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.255254] alg: No test for adler32 (adler32-zlib) [ 2205.176744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2205.250915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2205.277736] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2205.996691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2206.031770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.59@tcp [ 2206.049917] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2206.052749] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2206.055258] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2206, 0, 0 [ 2206.059242] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2206.061617] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2206.063994] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2206.066849] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2206, 0, 0 [ 2206.070985] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2206.073369] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2206.076325] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2206.080778] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2206, 0, 2256 [ 2206.084069] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2206.090423] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2206.094781] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2206.098431] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2206, 0, 2256 [ 2206.102908] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2211.360109] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 22:39:21 (1789267161) [ 2212.211800] LNet: 107712:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.219268] LNetError: 107712:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.236768] LNet: Removed LNI 192.168.201.59@tcp [ 2212.853623] Key type .llcrypt unregistered [ 2212.861220] Key type ._llcrypt unregistered [ 2219.377863] Key type ._llcrypt registered [ 2219.380492] Key type .llcrypt registered [ 2219.584372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2219.602764] alg: No test for adler32 (adler32-zlib) [ 2220.679717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2220.708522] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2227.953612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2239.034909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.128971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2239.181856] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2240.694828] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2240.859081] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2240.865221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2240.897813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2242.421855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2242.443591] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2254.403814] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 22:40:04 (1789267204) [ 2254.578983] LNet: 109202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.584307] LNetError: 109202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.599553] LNet: Removed LNI 192.168.201.59@tcp [ 2255.648621] LNet: Removed LNI 192.168.201.59@tcp1 [ 2255.653557] LNetError: 109202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2255.670367] LNetError: 109202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2255.689710] LNet: Removed LNI 10.1.2.3@tcp99 [ 2256.369992] Key type .llcrypt unregistered [ 2256.373642] Key type ._llcrypt unregistered [ 2262.614062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2262.620311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2265.316924] Key type ._llcrypt registered [ 2265.319602] Key type .llcrypt registered [ 2265.540150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2265.552515] alg: No test for adler32 (adler32-zlib) [ 2266.583361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.659819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.686619] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2267.662911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2267.708230] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2270.189789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2270.258720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2270.330358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2270.354248] LNetError: 109951:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2270.404365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2270.491253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2271.088392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2271.195868] LNet: 109971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.206811] LNetError: 109971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2271.214280] LNet: Removed LNI 10.1.2.3@tcp [ 2272.233340] LNet: Removed LNI 192.168.201.59@tcp [ 2273.361686] Key type .llcrypt unregistered [ 2273.375152] Key type ._llcrypt unregistered [ 2288.131935] Key type ._llcrypt registered [ 2288.136459] Key type .llcrypt registered [ 2289.386195] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 22:40:39 (1789267239) [ 2289.802499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.820768] alg: No test for adler32 (adler32-zlib) [ 2290.860602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.941788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.997500] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2291.885665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2291.961567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg159-client@tcp [ 2298.937442] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 22:40:49 (1789267249) [ 2299.084799] LNet: 111494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2299.092759] LNetError: 111494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2299.131199] LNet: Removed LNI 192.168.201.59@tcp [ 2299.891447] Key type .llcrypt unregistered [ 2299.893085] Key type ._llcrypt unregistered [ 2307.938737] Key type ._llcrypt registered [ 2307.939986] Key type .llcrypt registered [ 2308.113272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2308.123663] alg: No test for adler32 (adler32-zlib) [ 2309.131749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2309.184317] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2309.230871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2316.363598] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 22:41:06 (1789267266) [ 2318.133866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2319.812287] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 22:41:10 (1789267270) [ 2321.893589] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2323.614847] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 22:41:13 (1789267273) [ 2323.790653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2323.840124] LNet: 112921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2323.845607] LNetError: 112921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2323.858881] LNet: Removed LNI 192.168.201.59@tcp [ 2323.876167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.928877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.945875] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2324.770744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2324.842838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2331.561747] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 22:41:21 (1789267281) [ 2331.707397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2331.746532] LNet: 113626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2331.760029] LNetError: 113626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2331.790449] LNet: Removed LNI 192.168.201.59@tcp [ 2331.828711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.907278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2331.934289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2332.705360] Key type .llcrypt unregistered [ 2332.708846] Key type ._llcrypt unregistered [ 2344.558835] Key type ._llcrypt registered [ 2344.566990] Key type .llcrypt registered [ 2346.145400] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 22:41:35 (1789267295) [ 2346.580986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2346.602474] alg: No test for adler32 (adler32-zlib) [ 2347.540791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.601828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2347.627957] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2348.181500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2348.239757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2356.065876] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 22:41:45 (1789267305) [ 2356.208235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.254664] LNet: 115180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.272499] LNetError: 115180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.333916] LNet: Removed LNI 192.168.201.59@tcp [ 2356.362591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.448666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2356.493930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2364.084713] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 22:41:54 (1789267314) [ 2364.381418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2364.421076] LNet: 115763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2364.451676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.547396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2364.904795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2364.932074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2364.994172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2365.509089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2365.549570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2365.644833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.682150] LNet: 115797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2365.706903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.781105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2366.199330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2366.231182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2366.268283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2366.729884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2366.765753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2373.351872] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 22:42:03 (1789267323) [ 2373.462089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2373.508705] LNet: 116389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2373.564519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.645585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2373.721859] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2374.646197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2374.677595] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2375.778418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2375.837536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2376.165333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2376.202881] LNetError: 116676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2376.209470] LNet: Removed LNI 192.168.201.59@tcp [ 2376.225197] LNet: Removed LNI 192.168.201.59@tcp2 [ 2376.250132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.313098] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2376.319662] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2376.323560] LNet: 116682:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2376.898289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2377.112918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2378.207143] LNet: Removed LNI 192.168.201.59@tcp [ 2378.209176] LNetError: 116719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2378.221784] LNetError: 116719:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2378.256143] LNet: Removed LNI 192.168.201.59@tcp2 [ 2378.285359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2378.369949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2378.423782] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2379.435594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2379.464922] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2380.338235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2380.378978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2380.459223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2380.736946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.773147] LNet: 117004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.777252] LNet: 117004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2380.786795] LNetError: 117004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2381.855342] LNet: Removed LNI 192.168.201.59@tcp [ 2382.886876] LNet: Removed LNI 192.168.201.59@tcp2 [ 2382.928679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2383.018450] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2383.027554] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2383.036425] LNet: 117010:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2383.878798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2384.132483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2384.155554] LNet: Removed LNI 192.168.201.59@tcp [ 2385.192187] LNet: Removed LNI 192.168.201.59@tcp2 [ 2385.277312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2385.427192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2385.473144] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2386.205121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2386.233570] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2386.999976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2387.059810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2387.268621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.280578] LNetError: 117327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.286918] LNetError: 117327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2387.300661] LNet: Removed LNI 192.168.201.59@tcp [ 2387.311718] LNet: Removed LNI 192.168.201.59@tcp2 [ 2387.345576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.410833] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2387.422026] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2387.432550] LNet: 117333:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2387.920527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2388.123427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2389.215295] LNet: Removed LNI 192.168.201.59@tcp [ 2389.261097] LNet: Removed LNI 192.168.201.59@tcp2 [ 2389.306100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.457568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2389.493177] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2390.409847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2390.437439] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2391.647629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2391.707842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2391.770608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2392.053427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2392.083178] LNet: 117655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2392.103131] LNet: 117655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2393.119228] LNet: Removed LNI 192.168.201.59@tcp [ 2394.168379] LNet: Removed LNI 192.168.201.59@tcp2 [ 2394.239074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.336395] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2394.341747] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2394.346546] LNet: 117661:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2395.265149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2395.557273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2395.594924] LNetError: 117699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2395.605184] LNetError: 117699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2395.619436] LNet: Removed LNI 192.168.201.59@tcp [ 2396.652726] LNet: Removed LNI 192.168.201.59@tcp2 [ 2396.703353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2396.818529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2396.864621] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2397.821702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2397.848979] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2398.620127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2398.657192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2398.722265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2399.147689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.223876] LNet: Removed LNI 192.168.201.59@tcp [ 2400.246465] LNet: Removed LNI 192.168.201.59@tcp2 [ 2400.304314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.401689] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2400.409938] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2400.419328] LNet: 118002:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2401.347265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2401.843071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2402.911222] LNet: Removed LNI 192.168.201.59@tcp [ 2402.923958] LNet: Removed LNI 192.168.201.59@tcp2 [ 2402.989223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.093817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2403.142899] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2404.106121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2404.133101] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2405.015387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2405.053837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2405.087985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2405.150877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2405.503423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.559140] LNet: Removed LNI 192.168.201.59@tcp [ 2407.592985] LNet: Removed LNI 192.168.201.59@tcp2 [ 2407.698877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2407.792283] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2407.799966] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2408.220806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2416.126502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2418.791554] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 22:42:48 (1789267368) [ 2419.055647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2419.095134] LNet: 119118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.101515] LNet: 119118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2419.112485] LNetError: 119118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2419.121783] LNetError: 119118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2419.129060] LNet: Removed LNI 192.168.201.59@tcp [ 2420.197673] LNet: Removed LNI 192.168.201.59@tcp2 [ 2420.294368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2420.675666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.729606] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2421.880681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2421.912677] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2423.270735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2423.303535] LNet: Removed LNI 10.1.2.3@tcp [ 2423.332580] LNet: Removed LNI 192.168.201.59@tcp [ 2423.368286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2423.561882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2423.664787] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2423.674947] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2424.303898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2424.341074] LNet: Removed LNI 192.168.201.59@tcp [ 2424.368881] LNet: Removed LNI 10.1.2.3@tcp [ 2431.688369] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 22:43:01 (1789267381) [ 2431.997377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2432.038848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2432.125796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2432.163725] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2432.952668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2432.989682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.022757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.061794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.107655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.142694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.188635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.234704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.287474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.330924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.377994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.59@tcp [ 2433.538319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2440.777363] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 22:43:10 (1789267390) [ 2440.875598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.958825] LNet: Removed LNI 192.168.201.59@tcp [ 2441.999447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.132051] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2442.336623] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2442.344233] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2442.352069] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2444.732697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2444.772534] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2445.501567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2445.532329] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2447.561956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2447.614595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2447.671885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.59@tcp -m PUT -e local_timeout -r 1 [ 2447.726681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2447.787259] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2447.798057] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2448.025737] LNetError: 120804:0:(lib-move.c:4801:lnet_parse()) 192.168.201.59@tcp, src 192.168.201.59@tcp: Dropping PUT (error -108 looking up sender) [ 2449.057965] LNet: Removed LNI 10.1.2.3@tcp [ 2450.086829] LNet: Removed LNI 192.168.201.59@tcp [ 2450.458837] Key type .llcrypt unregistered [ 2450.461540] Key type ._llcrypt unregistered [ 2464.552513] Key type ._llcrypt registered [ 2464.555703] Key type .llcrypt registered [ 2465.835631] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 22:43:35 (1789267415) [ 2466.235119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2466.247958] alg: No test for adler32 (adler32-zlib) [ 2467.209326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2467.539468] LNet: 122112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2468.228568] Key type .llcrypt unregistered [ 2468.230213] Key type ._llcrypt unregistered [ 2475.202344] Key type ._llcrypt registered [ 2475.203563] Key type .llcrypt registered [ 2475.422678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2475.441753] alg: No test for adler32 (adler32-zlib) [ 2476.329510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2476.359691] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2484.453580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2484.713211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2485.301400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2485.546463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2485.578852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2485.605499] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2485.624209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2485.655637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2485.696296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2485.721995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2485.743487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2485.772907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2485.800990] LNet: 122560:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.826418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2485.881880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2485.918460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2485.960962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2486.005959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2486.038290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2486.593804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2486.823695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2486.853506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2486.880407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2486.905955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2486.929371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2486.958192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2486.987779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2487.013284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2487.026693] LNet: 122677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2487.041792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2487.065679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2487.094136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2487.124454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2487.151151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2487.182970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2487.743488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2487.966847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2488.005243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2488.031254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2488.074518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2488.122518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2488.149818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2488.203570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2488.261402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2488.291790] LNet: 122794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2488.330638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2488.394319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2488.461040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2488.502590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2488.537354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2488.582996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2489.377055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2489.632098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2489.688222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2489.750547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2489.810735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2489.861459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2489.897810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2489.948613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2490.018576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2490.060539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2490.115487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2490.158074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2490.212831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2490.271326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2490.320896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2490.614834] LNet: 122958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2490.625515] LNetError: 122958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2490.639902] LNet: Removed LNI 192.168.201.59@tcp [ 2491.159364] Key type .llcrypt unregistered [ 2491.164485] Key type ._llcrypt unregistered [ 2510.005658] Key type ._llcrypt registered [ 2510.008468] Key type .llcrypt registered [ 2512.090836] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 22:44:21 (1789267461) [ 2512.795934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2512.827125] alg: No test for adler32 (adler32-zlib) [ 2513.848175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2514.439508] LNet: 123844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2515.272842] Key type .llcrypt unregistered [ 2515.276144] Key type ._llcrypt unregistered [ 2521.608683] Key type ._llcrypt registered [ 2521.610299] Key type .llcrypt registered [ 2521.823497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2521.853822] alg: No test for adler32 (adler32-zlib) [ 2522.765632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2522.777782] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2529.831273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.159@tcp [ 2538.671537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2538.701595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2538.729600] LNet: Added LNI 192.168.201.59@tcp1 [8/256/0/180] [ 2539.511101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2540.142778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.159@tcp [ 2540.191288] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.159@tcp [ 2540.399690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2540.432964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2540.461389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2540.492674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2540.523582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2540.556384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2540.583917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2540.620505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2540.640816] LNet: 124494:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2540.660707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2540.688198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2540.728687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2540.763440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2540.801440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2540.838923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2541.481494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2541.713385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2541.743263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2541.778262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2541.813105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2541.835374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2541.889131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2541.924538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2541.958895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2541.981960] LNet: 124611:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2542.015952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2542.071994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2542.118516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2542.168048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2542.223858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2542.276694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.830702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.159@tcp [ 2543.086634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2543.124170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2543.159574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2543.189767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2543.218862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2543.246981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2543.270830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2543.289162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2543.299594] LNet: 124728:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2543.312282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2543.336482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.364869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.395323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.424789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.456654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2544.060472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2544.283810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2544.308124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2544.330928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2544.356845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2544.377847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2544.400677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2544.422264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2544.444113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2544.465813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.492305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.515232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.536300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2544.562851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.596507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2549.496969] LNet: 124926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.502948] LNetError: 124926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2549.509699] LNet: Removed LNI 192.168.201.59@tcp [ 2549.518790] LNet: Removed LNI 192.168.201.59@tcp1 [ 2549.982771] Key type .llcrypt unregistered [ 2549.984378] Key type ._llcrypt unregistered [ 2567.512100] Key type ._llcrypt registered [ 2567.518227] Key type .llcrypt registered [ 2569.559964] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 22:45:18 (1789267518) [ 2570.303314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2570.326850] alg: No test for adler32 (adler32-zlib) [ 2571.326652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2571.433790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2571.478096] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2572.395088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2572.415139] LNet: Added LNI 192.168.201.59@tcp2 [8/256/0/180] [ 2573.370914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.59@tcp [ 2573.456741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2574.495636] LNet: Removed LNI 192.168.201.59@tcp [ 2582.005396] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 22:45:32 (1789267532) [ 2583.734963] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2585.423980] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 22:45:35 (1789267535) [ 2593.972209] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 22:45:44 (1789267544) [ 2601.704456] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 22:45:51 (1789267551) [ 2609.279964] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 22:45:59 (1789267559) [ 2617.645595] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 22:46:07 (1789267567) [ 2624.927768] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 22:46:15 (1789267575) [ 2633.002071] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 22:46:22 (1789267582) [ 2640.641616] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 22:46:30 (1789267590) [ 2648.156813] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 22:46:38 (1789267598) [ 2655.736895] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 22:46:45 (1789267605) [ 2663.506793] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 22:46:53 (1789267613) [ 2671.168494] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 22:47:01 (1789267621) [ 2678.046614] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 22:47:08 (1789267628) [ 2686.175151] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 22:47:15 (1789267635) [ 2693.925421] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 22:47:24 (1789267644) [ 2703.389591] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 22:47:33 (1789267653) [ 2711.921593] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 22:47:41 (1789267661) [ 2721.419777] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 22:47:51 (1789267671) [ 2721.625830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2721.669376] LNet: 136615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2721.675198] LNetError: 136615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.702731] LNet: Removed LNI 192.168.201.59@tcp2 [ 2721.758358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2721.906682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2721.929285] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2723.534410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2723.568693] LNet: 136761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2723.575667] LNetError: 136761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2723.602712] LNet: Removed LNI 192.168.201.59@tcp [ 2723.637763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2723.703070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2723.724804] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2732.941394] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 22:48:02 (1789267682) [ 2741.295162] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 22:48:11 (1789267691) [ 2750.809903] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 22:48:20 (1789267700) [ 2761.660463] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 22:48:31 (1789267711) [ 2770.134721] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 22:48:40 (1789267720) [ 2778.845947] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 22:48:48 (1789267728) [ 2789.035580] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 22:48:58 (1789267738) [ 2797.993266] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 22:49:07 (1789267747) [ 2807.343854] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 22:49:16 (1789267756) [ 2815.386305] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 22:49:25 (1789267765) [ 2823.133736] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 22:49:33 (1789267773) [ 2830.548909] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 22:49:40 (1789267780) [ 2838.128890] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2534 sec ========== 22:49:47 (1789267787) [ 2838.512299] LNet: 144231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2838.518521] LNetError: 144231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2838.539336] LNet: Removed LNI 192.168.201.59@tcp [ 2839.235699] Key type .llcrypt unregistered [ 2839.238490] Key type ._llcrypt unregistered [ 2852.065608] Key type ._llcrypt registered [ 2852.073278] Key type .llcrypt registered [ 2852.803249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2852.822214] alg: No test for adler32 (adler32-zlib) [ 2854.000918] Lustre: Lustre: Build Version: 2.17.57_105_g2595dd0 [ 2854.424377] LNet: Added LNI 192.168.201.59@tcp [8/256/0/180] [ 2856.127599] Key type lgssc registered [ 2857.872473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2928.829795] Lustre: Mounted lustre-client - version 2.17.57_105_g2595dd0 [ 2934.309643] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2942.019753] Lustre: Unmounted lustre-client [ 2976.063892] Key type lgssc unregistered [ 2976.429642] LNet: 147186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2976.436058] LNetError: 147186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2976.464109] LNet: Removed LNI 192.168.201.59@tcp [ 2977.417428] Key type .llcrypt unregistered [ 2977.424803] Key type ._llcrypt unregistered