[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 439190563 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.003069] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010008] pid_max: default: 32768 minimum: 301 [ 0.011169] LSM: Security Framework initializing [ 0.012069] Yama: becoming mindful. [ 0.013045] SELinux: Initializing. [ 0.014090] *** VALIDATE selinux *** [ 0.022738] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026845] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027163] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031135] *** VALIDATE tmpfs *** [ 0.033358] *** VALIDATE proc *** [ 0.034283] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036296] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040014] Speculative Store Bypass: Vulnerable [ 0.043502] debug: unmapping init [mem 0xffffffff9fc59000-0xffffffff9fc60fff] [ 0.046202] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047674] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051011] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055347] rcu: Hierarchical SRCU implementation. [ 0.057513] smp: Bringing up secondary CPUs ... [ 0.058635] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062442] smp: Brought up 1 node, 4 CPUs [ 0.063803] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130015] node 0 deferred pages initialised in 63ms [ 0.132357] devtmpfs: initialized [ 0.133233] x86/mm: Memory block size: 128MB [ 0.135244] gcov: version magic: 0x41383552 [ 0.137161] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.139089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142256] pinctrl core: initialized pinctrl subsystem [ 0.144120] [ 0.144416] ************************************************************* [ 0.146016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149016] ** ** [ 0.151007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153012] ** ** [ 0.155011] ** This means that this kernel is built to expose internal ** [ 0.157011] ** IOMMU data structures, which may compromise security on ** [ 0.159011] ** your system. ** [ 0.162013] ** ** [ 0.164012] ** If you see this message and you are not debugging the ** [ 0.166013] ** kernel, report this immediately to your vendor! ** [ 0.169014] ** ** [ 0.171012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173010] ************************************************************* [ 0.175679] NET: Registered protocol family 16 [ 0.177617] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.179091] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.182064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186038] cpuidle: using governor menu [ 0.188218] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190481] PCI: Using configuration type 1 for base access [ 0.192271] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.202047] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207141] cryptd: max_cpu_qlen set to 1000 [ 0.208222] ACPI: Added _OSI(Module Device) [ 0.210015] ACPI: Added _OSI(Processor Device) [ 0.211012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217492] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221551] ACPI: Interpreter enabled [ 0.223069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225015] ACPI: Using IOAPIC for interrupt routing [ 0.226125] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241030] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.243018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.245092] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248036] acpiphp: Slot [2] registered [ 0.249090] acpiphp: Slot [5] registered [ 0.250069] acpiphp: Slot [6] registered [ 0.251041] acpiphp: Slot [3] registered [ 0.252023] acpiphp: Slot [4] registered [ 0.252993] acpiphp: Slot [7] registered [ 0.254063] acpiphp: Slot [8] registered [ 0.255087] acpiphp: Slot [9] registered [ 0.257097] acpiphp: Slot [10] registered [ 0.258079] acpiphp: Slot [11] registered [ 0.259073] acpiphp: Slot [12] registered [ 0.260072] acpiphp: Slot [13] registered [ 0.261093] acpiphp: Slot [14] registered [ 0.262084] acpiphp: Slot [15] registered [ 0.264103] acpiphp: Slot [16] registered [ 0.265086] acpiphp: Slot [17] registered [ 0.266087] acpiphp: Slot [18] registered [ 0.268113] acpiphp: Slot [19] registered [ 0.269090] acpiphp: Slot [20] registered [ 0.270091] acpiphp: Slot [21] registered [ 0.272126] acpiphp: Slot [22] registered [ 0.273090] acpiphp: Slot [23] registered [ 0.274131] acpiphp: Slot [24] registered [ 0.276095] acpiphp: Slot [25] registered [ 0.277087] acpiphp: Slot [26] registered [ 0.278120] acpiphp: Slot [27] registered [ 0.280107] acpiphp: Slot [28] registered [ 0.281110] acpiphp: Slot [29] registered [ 0.283090] acpiphp: Slot [30] registered [ 0.284097] acpiphp: Slot [31] registered [ 0.285057] PCI host bridge to bus 0000:00 [ 0.287022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.289033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.291022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.293027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.295024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.298026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.300175] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303050] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305727] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.312528] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.316300] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.318015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.320012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.322014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.323547] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.325759] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.328041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.331827] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.334010] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.342010] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.346013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.352699] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.359016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.365017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.379037] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.388828] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.394014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.399015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.412018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.421344] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.423297] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.424321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.426341] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.428202] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.432076] iommu: Default domain type: Passthrough [ 0.434444] SCSI subsystem initialized [ 0.435154] ACPI: bus type USB registered [ 0.436066] usbcore: registered new interface driver usbfs [ 0.438076] usbcore: registered new interface driver hub [ 0.439060] usbcore: registered new device driver usb [ 0.440127] pps_core: LinuxPPS API ver. 1 registered [ 0.442011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.445068] PTP clock support registered [ 0.446148] EDAC MC: Ver: 3.0.0 [ 0.448114] PCI: Using ACPI for IRQ routing [ 0.449000] NetLabel: Initializing [ 0.450006] NetLabel: domain hash size = 128 [ 0.451006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.452060] NetLabel: unlabeled traffic allowed by default [ 0.453114] vgaarb: loaded [ 0.454151] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.455007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.461354] clocksource: Switched to clocksource kvm-clock [ 0.556521] VFS: Disk quotas dquot_6.6.0 [ 0.557993] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.561101] *** VALIDATE ramfs *** [ 0.562340] *** VALIDATE hugetlbfs *** [ 0.563907] pnp: PnP ACPI init [ 0.566219] pnp: PnP ACPI: found 6 devices [ 0.586422] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.589367] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.591560] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.593443] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.595625] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.597473] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.599755] NET: Registered protocol family 2 [ 0.602141] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.606526] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.609191] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.613500] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.616753] TCP: Hash tables configured (established 65536 bind 65536) [ 0.619030] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.622244] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.624842] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.627668] NET: Registered protocol family 1 [ 0.629254] RPC: Registered named UNIX socket transport module. [ 0.630860] RPC: Registered udp transport module. [ 0.631777] RPC: Registered tcp transport module. [ 0.632747] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.634017] NET: Registered protocol family 44 [ 0.635124] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.636335] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.637440] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.638841] PCI: CLS 0 bytes, default 64 [ 0.639932] Unpacking initramfs... [ 1.938560] debug: unmapping init [mem 0xffff8db2fcc64000-0xffff8db2fffcffff] [ 1.941828] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.944338] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.947430] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.428903] Initialise system trusted keyrings [ 2.430358] Key type blacklist registered [ 2.432204] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.440882] zbud: loaded [ 2.443759] *** VALIDATE nfs *** [ 2.445115] *** VALIDATE nfs4 *** [ 2.446500] pstore: using deflate compression [ 2.449571] Platform Keyring initialized [ 2.537912] NET: Registered protocol family 38 [ 2.539292] Key type asymmetric registered [ 2.540266] Asymmetric key parser 'x509' registered [ 2.541413] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.543405] io scheduler mq-deadline registered [ 2.544369] io scheduler kyber registered [ 2.549488] io scheduler bfq registered [ 2.550843] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.552964] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.555319] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.558213] ACPI: Power Button [PWRF] [ 2.563819] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.571702] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.588777] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.618189] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.645086] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.650728] Non-volatile memory driver v1.3 [ 2.652770] Linux agpgart interface v0.103 [ 2.681299] virtio_blk virtio1: [vda] 134360 512-byte logical blocks (68.8 MB/65.6 MiB) [ 2.684389] vda: detected capacity change from 0 to 68792320 [ 2.699738] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.702843] vdb: detected capacity change from 0 to 1073741824 [ 2.708643] libphy: Fixed MDIO Bus: probed [ 2.715528] usbcore: registered new interface driver usbserial_generic [ 2.717891] usbserial: USB Serial support registered for generic [ 2.720276] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.724276] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.726145] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.728623] mousedev: PS/2 mouse device common for all mice [ 2.730268] rtc_cmos 00:05: RTC can wake from S4 [ 2.731924] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.733393] rtc_cmos 00:05: registered as rtc0 [ 2.734969] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.738312] intel_pstate: CPU model not supported [ 2.741390] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.745276] hid: raw HID events driver (C) Jiri Kosina [ 2.747779] usbcore: registered new interface driver usbhid [ 2.749943] usbhid: USB HID core driver [ 2.751725] drop_monitor: Initializing network drop monitor service [ 2.751781] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.755182] Initializing XFRM netlink socket [ 2.759217] NET: Registered protocol family 10 [ 2.761826] Segment Routing with IPv6 [ 2.762935] NET: Registered protocol family 17 [ 2.764482] mpls_gso: MPLS GSO support [ 2.770745] RAS: Correctable Errors collector initialized. [ 2.772551] AVX version of gcm_enc/dec engaged. [ 2.774171] AES CTR mode by8 optimization enabled [ 2.834997] sched_clock: Marking stable (2834974767, 0)->(3714111693, -879136926) [ 2.839510] registered taskstats version 1 [ 2.841626] Loading compiled-in X.509 certificates [ 2.842845] zswap: loaded using pool lzo/zbud [ 2.859135] Key type big_key registered [ 2.869899] Key type encrypted registered [ 2.871749] ima: No TPM chip found, activating TPM-bypass! [ 2.874065] ima: Allocated hash algorithm: sha1 [ 2.874981] ima: No architecture policies found [ 2.876516] evm: Initialising EVM extended attributes: [ 2.877600] evm: security.selinux [ 2.878381] evm: security.ima [ 2.878996] evm: security.capability [ 2.879941] evm: HMAC attrs: 0x1 [ 2.881699] rtc_cmos 00:05: setting system clock to 2026-01-30 06:22:04 UTC (1769754124) [ 2.885671] debug: unmapping init [mem 0xffffffffa0c03000-0xffffffffa0dfffff] [ 2.887679] debug: unmapping init [mem 0xffffffff9f982000-0xffffffff9fc58fff] [ 2.895142] Write protecting the kernel read-only data: 28672k [ 2.897849] debug: unmapping init [mem 0xffffffff9e003000-0xffffffff9e1fffff] [ 2.899810] debug: unmapping init [mem 0xffffffff9e914000-0xffffffff9e9fffff] [ 2.927847] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 2.935593] systemd[1]: Detected virtualization kvm. [ 2.937453] systemd[1]: Detected architecture x86-64. [ 2.939295] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 2.961838] systemd[1]: No hostname configured. [ 2.962983] systemd[1]: Set hostname to . [ 2.965236] random: systemd: uninitialized urandom read (16 bytes read) [ 2.967580] systemd[1]: Initializing machine ID from random generator. [ 3.009202] random: ln: uninitialized urandom read (6 bytes read) [ 3.073323] random: systemd: uninitialized urandom read (16 bytes read) [ 3.076547] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.080530] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.087767] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.681895] device-mapper: uevent: version 1.0.3 [ 3.684143] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 4.208273] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.352454] virtio_net virtio0 ens2: renamed from eth0 [ 4.420608] scsi host0: ata_piix [ 4.440274] scsi host1: ata_piix [ 4.441744] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.444470] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.888990] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.384929] random: crng init done [ 9.386236] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.753699] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.833475] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.056566] SELinux: Disabled at runtime. [ 11.115320] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.124483] systemd[1]: Detected virtualization kvm. [ 11.126460] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.567605] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.570895] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.575785] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.579620] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.583420] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.591997] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.597566] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ 11.691477] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.016990] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.355211] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.414409] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.454278] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.471559] EDAC sbridge: Ver: 1.1.2 [ 13.467090] Key type dns_resolver registered [ 13.768309] NFS: Registering the id_resolver key type [ 13.770874] Key type id_resolver registered [ 13.772704] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg126-client login: [ 44.968863] libcfs: loading out-of-tree module taints kernel. [ 45.039628] Key type ._llcrypt registered [ 45.057617] Key type .llcrypt registered [ 45.256879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.262296] alg: No test for adler32 (adler32-zlib) [ 46.327868] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 46.675691] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 48.311169] Key type lgssc registered [ 48.901497] Lustre: Echo OBD driver; http://www.lustre.org/ [ 167.217863] Lustre: Mounted lustre-client [ 172.653664] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 180.580605] hrtimer: interrupt took 4400912 ns [ 192.298849] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing check_logdir /tmp/testlogs/ [ 192.991255] Lustre: lustre-OST0000-osc-ffff8db34a653000: disconnect after 23s idle [ 197.199060] Lustre: DEBUG MARKER: oleg126-client.virtnet: executing yml_node [ 201.712204] Lustre: DEBUG MARKER: Client: 2.17.50.86 [ 204.267316] Lustre: DEBUG MARKER: MDS: 2.17.50.86 [ 206.423702] Lustre: DEBUG MARKER: OSS: 2.17.50.86 [ 207.812246] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jan 30 01:25:28 EST 2026 [ 223.959958] LustreError: 7921:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8db34a653000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 223.969784] LustreError: 7921:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 224.001503] Lustre: Unmounted lustre-client [ 252.762577] Key type lgssc unregistered [ 253.064851] LNet: 8369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 253.073378] LNetError: 8369:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 254.135116] LNet: Removed LNI 192.168.201.26@tcp [ 254.999167] Key type .llcrypt unregistered [ 255.000963] Key type ._llcrypt unregistered [ 281.006514] Key type ._llcrypt registered [ 281.008856] Key type .llcrypt registered [ 281.203727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 281.222682] alg: No test for adler32 (adler32-zlib) [ 282.128922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 282.183348] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 282.209423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 286.274630] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 286.627170] LNet: 9245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.639990] LNetError: 9245:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 286.687589] LNet: Removed LNI 192.168.201.26@tcp [ 287.411605] Key type .llcrypt unregistered [ 287.415901] Key type ._llcrypt unregistered [ 293.431942] Key type ._llcrypt registered [ 293.435456] Key type .llcrypt registered [ 294.870868] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:26:54 (1769754414) [ 295.345227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 295.366994] alg: No test for adler32 (adler32-zlib) [ 296.364693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.453723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.470594] LNet: 9652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.489893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.538960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.554608] LNet: 9659:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 303.915505] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:27:04 (1769754424) [ 304.068967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.095576] LNet: 10234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.119118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 304.226568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 311.305521] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:27:11 (1769754431) [ 311.474030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.501137] LNet: 10823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.522240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.601269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 317.903441] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:27:18 (1769754438) [ 318.389365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.418781] LNet: 11407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 318.449208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.516228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 325.816719] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:27:25 (1769754445) [ 325.986508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.015918] LNet: 11990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.081096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.227879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 326.760493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 333.854826] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:27:33 (1769754453) [ 334.050903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 334.086568] LNet: 12580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 334.130173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 334.226115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 334.830303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 335.862815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 346.745890] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:27:46 (1769754466) [ 346.870489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 346.991405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.230377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 356.717750] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:27:56 (1769754476) [ 357.110603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.157818] LNet: 13767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.164756] LNet: 13767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 357.200442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.289637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 357.326120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 357.816703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 357.881812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 358.341231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 358.400695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 359.191890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 359.246549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.759845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 359.822350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.500721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 360.539220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.134103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 361.188408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.649300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 361.699396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.034204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 362.063641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 362.444634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 362.475375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 362.987705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 363.013372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 363.273312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 363.307725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 363.782884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 363.830812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 371.891452] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:28:11 (1769754491) [ 372.042903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.121473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.236588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 372.266968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 379.492885] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:28:19 (1769754499) [ 379.929356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.005219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.145209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 380.191428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 387.731244] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:28:27 (1769754507) [ 388.010256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.080975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.204288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 388.281625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 396.576843] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:28:36 (1769754516) [ 396.699225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.722604] LNet: 16201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.725837] LNet: 16201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 396.756420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.864776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 396.907671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 403.763347] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:28:43 (1769754523) [ 404.084756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.117559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.172861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 404.199572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 412.387517] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:28:52 (1769754532) [ 412.521804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.577086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.753533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 412.811631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 421.121279] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:29:00 (1769754540) [ 421.490096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.535649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.625440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 422.202574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 429.714384] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:29:10 (1769754550) [ 429.921714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.981990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.222475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 430.456168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 436.953808] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:29:17 (1769754557) [ 437.170149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.212097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 437.335678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 437.553679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 445.080676] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:29:25 (1769754565) [ 445.168957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.250526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.782051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.027453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 452.396181] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:29:32 (1769754572) [ 452.593325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.630300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.684929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.731744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.154665] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:29:40 (1769754580) [ 460.390485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.471833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.573788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 460.616278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.291291] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:29:48 (1769754588) [ 468.666706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.697835] LNet: 21651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 468.700803] LNet: 21651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 468.716354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.789863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 475.946955] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:29:55 (1769754595) [ 476.070942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.161324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.308640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 483.169743] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:30:03 (1769754603) [ 483.234726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.280610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.405513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 484.213354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 491.458858] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:30:11 (1769754611) [ 491.781611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.817735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.899691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 491.941487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 498.898748] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:30:19 (1769754619) [ 499.217731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.254488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.332368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 499.365630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 508.037421] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:30:27 (1769754627) [ 508.605761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.702616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.850079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 508.939086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 517.129925] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:30:37 (1769754637) [ 517.209227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.274452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.460279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 517.499589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.333591] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:30:45 (1769754645) [ 525.418396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.495202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.583930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 525.705277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 525.726719] LNetError: 25770:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 525.767473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 525.798174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 532.407284] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:30:52 (1769754652) [ 532.718624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.787469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.899840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.a8v26BOuiG [ 532.990435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.a8v26BOuiG [ 533.038208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.a8v26BOuiG [ 539.783135] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:31:00 (1769754660) [ 540.124553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.168920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.234412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 540.277526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 540.339923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 547.180913] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:31:07 (1769754667) [ 547.279497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.347979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.468409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 550.070456] LNet: Message forwarding will be enabled in 2s [ 553.183230] LNet: Message forwarding enabled [ 553.601937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 553.664679] LNet: Message forwarding will stop in 2s to 6s [ 558.415575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 559.247196] LNet: Message forwarding disabled [ 560.800691] LNet: Message forwarding will be enabled in 2s [ 564.386147] LNet: Message forwarding enabled [ 570.214253] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:31:29 (1769754689) [ 570.362145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.810395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.912044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 579.343725] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:31:39 (1769754699) [ 579.438490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.494808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.597555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 579.639509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 579.675453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 579.699861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 579.739339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 579.779587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 579.827916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 579.881754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 579.928372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 579.971420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 580.013841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 580.056276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 580.078380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 580.108756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 580.139676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 580.258287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 580.325515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 580.377491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 580.431529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 580.454877] LNetError: 28841:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 587.611856] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:31:47 (1769754707) [ 587.728664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.811398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.964623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 594.626988] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:31:55 (1769754715) [ 596.910244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg126-server [ 598.741118] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:31:59 (1769754719) [ 599.014472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 599.043276] LNet: 30343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.048472] LNet: 30343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 599.073545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 599.142384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 599.171791] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 600.579792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[27-31]@tcp [ 600.617816] LNet: 30558:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 600.685611] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 600.701719] LNetError: 30347:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.27@tcp has gone from up to down [ 601.715337] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 601.721250] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 602.726130] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 602.752175] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 604.775409] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 604.788621] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 607.842500] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:32:07 (1769754727) [ 608.874090] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.27@tcp is being used as a gateway but routing feature is not turned on [ 608.884963] LNetError: 30347:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 609.664226] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg126-server [ 611.362366] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:32:11 (1769754731) [ 611.664836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.682881] LNetError: 31492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.717514] LNet: Removed LNI 192.168.201.26@tcp [ 612.771221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.857523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 612.897824] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 613.637576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[27-32/2]@tcp [ 613.651893] LNet: 31610:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 613.657251] LNet: 31610:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 613.673145] LNetError: 31497:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.31@tcp has gone from up to down [ 613.682470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[27-32/2]@tcp [ 613.690804] LNetError: 31497:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 619.531783] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:32:19 (1769754739) [ 619.642980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.730902] LNetError: 32196:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 619.766134] LNet: Removed LNI 192.168.201.26@tcp [ 619.833200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.945633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 620.004573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.037736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.150377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.214800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.248883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.346656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 620.489218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.533177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.618074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 620.698794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.737175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.796142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 620.891514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.923259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.043722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.133307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.190630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.243103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 621.300490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.336590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.399386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.460630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.493282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.552597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 621.639567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.698428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.769038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 621.864693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.927445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.030555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 622.128195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.178478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.244756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.162611] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:32:29 (1769754749) [ 631.263312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg126-server [ 632.886287] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:32:33 (1769754753) [ 634.927772] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg126-server [ 636.452796] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:32:36 (1769754756) [ 637.556767] Key type .llcrypt unregistered [ 637.559772] Key type ._llcrypt unregistered [ 644.543136] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 644.678975] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 644.683961] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 644.703312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 647.317654] Key type ._llcrypt registered [ 647.326090] Key type .llcrypt registered [ 647.485416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 647.497845] alg: No test for adler32 (adler32-zlib) [ 648.393359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.462386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.485143] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 649.223534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 649.235659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 651.057618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 651.072804] LNet: Removed LNI 10.1.2.3@tcp [ 651.196336] LNet: 34321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.202526] LNetError: 34321:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.215538] LNet: Removed LNI 192.168.201.26@tcp [ 651.801736] Key type .llcrypt unregistered [ 651.805027] Key type ._llcrypt unregistered [ 663.475468] Key type ._llcrypt registered [ 663.478404] Key type .llcrypt registered [ 664.413397] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:33:04 (1769754784) [ 665.058172] Key type .llcrypt unregistered [ 665.064125] Key type ._llcrypt unregistered [ 671.861627] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.080741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.086194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.093577] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 674.974113] Key type ._llcrypt registered [ 674.976709] Key type .llcrypt registered [ 675.150196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.187096] alg: No test for adler32 (adler32-zlib) [ 676.232906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.351251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.386394] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 677.241194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.260141] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 680.137763] LNet: 35810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 680.153741] LNetError: 35810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 680.155887] LNet: Removed LNI 192.168.201.26@tcp [ 680.168533] LNet: Removed LNI 10.1.2.3@tcp [ 680.897581] Key type .llcrypt unregistered [ 680.898821] Key type ._llcrypt unregistered [ 694.764788] Key type ._llcrypt registered [ 694.766662] Key type .llcrypt registered [ 696.247656] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:33:36 (1769754816) [ 697.052633] Key type .llcrypt unregistered [ 697.054677] Key type ._llcrypt unregistered [ 706.821406] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 706.957818] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 706.976093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 706.992305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 710.426567] Key type ._llcrypt registered [ 710.430842] Key type .llcrypt registered [ 710.687864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 710.699500] alg: No test for adler32 (adler32-zlib) [ 711.804118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 711.891532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 711.918265] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 714.932416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 714.970778] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 717.466515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 717.499224] LNet: Removed LNI 10.1.2.3@tcp [ 717.508689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 717.530378] LNet: Removed LNI 10.1.2.31@tcp [ 717.546524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.566404] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.377453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 718.405257] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 719.645808] LNet: 37742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.658276] LNetError: 37742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 719.673215] LNet: Removed LNI 10.1.2.3@tcp [ 719.694656] LNet: Removed LNI 10.1.2.31@tcp [ 720.710876] Key type .llcrypt unregistered [ 720.716530] Key type ._llcrypt unregistered [ 735.142868] Key type ._llcrypt registered [ 735.144387] Key type .llcrypt registered [ 736.575744] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:34:16 (1769754856) [ 737.029612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 737.046950] alg: No test for adler32 (adler32-zlib) [ 737.951140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.988500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 738.003752] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 738.580361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.598410] LNet: 38690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.603342] LNetError: 38690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.617250] LNet: Removed LNI 192.168.201.26@tcp [ 738.636073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.671886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.26@tcp [ 738.697870] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 739.555626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.26@tcp [ 739.581384] LNet: Removed LNI 192.168.201.26@tcp [ 739.628803] LNet: 38825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.261430] Key type .llcrypt unregistered [ 740.267725] Key type ._llcrypt unregistered [ 754.319641] Key type ._llcrypt registered [ 754.332807] Key type .llcrypt registered [ 756.556779] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:34:35 (1769754875) [ 758.885979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg126-server [ 761.309463] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:34:40 (1769754880) [ 762.002555] Key type .llcrypt unregistered [ 762.003845] Key type ._llcrypt unregistered [ 768.757588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.003524] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.016691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.047423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.804245] Key type ._llcrypt registered [ 771.806709] Key type .llcrypt registered [ 772.035021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.089405] alg: No test for adler32 (adler32-zlib) [ 773.149280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.231327] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 773.867503] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 776.028655] LNet: 40649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.033972] LNetError: 40649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.039757] LNet: Removed LNI 192.168.201.26@tcp1 [ 776.049848] LNet: Removed LNI 10.1.2.3@tcp2 [ 776.508199] Key type .llcrypt unregistered [ 776.510759] Key type ._llcrypt unregistered [ 788.333908] Key type ._llcrypt registered [ 788.335923] Key type .llcrypt registered [ 789.912458] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:35:09 (1769754909) [ 790.447436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 790.474272] alg: No test for adler32 (adler32-zlib) [ 791.544985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.594693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.613325] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 792.176494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.190703] LNet: 41613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.200757] LNetError: 41613:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.215832] LNet: Removed LNI 192.168.201.26@tcp [ 792.264127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.351190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 792.366745] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 800.372761] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:35:20 (1769754920) [ 800.656154] LNet: 42198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.681817] LNetError: 42198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 800.725422] LNet: Removed LNI 192.168.201.26@tcp [ 801.764379] Key type .llcrypt unregistered [ 801.765794] Key type ._llcrypt unregistered [ 811.652466] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.020695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.035176] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.049061] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.838433] Key type ._llcrypt registered [ 814.840547] Key type .llcrypt registered [ 815.034416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 815.064887] alg: No test for adler32 (adler32-zlib) [ 816.026911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.088451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 816.113859] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 816.820138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 816.859437] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 819.235338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.262206] LNet: 42938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.270964] LNetError: 42938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.280850] LNet: Removed LNI 192.168.201.26@tcp [ 819.297291] LNet: Removed LNI 10.1.2.3@tcp [ 819.361563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.461859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 819.504620] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 819.511691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.227430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.255419] LNet: 42965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.267980] LNetError: 42965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.274754] LNetError: 42965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 820.280746] LNet: Removed LNI 192.168.201.26@tcp [ 820.293929] LNet: Removed LNI 10.1.2.3@tcp [ 820.327377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.381159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 820.397875] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 820.401170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.710990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 820.722661] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 821.651792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 821.675653] LNet: 43127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 821.681149] LNetError: 43127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 821.686825] LNetError: 43127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 821.692816] LNet: Removed LNI 192.168.201.26@tcp [ 821.694602] LNet: Removed LNI 10.1.2.3@tcp [ 821.703246] LNet: Removed LNI 192.168.201.26@tcp2 [ 821.724368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 821.849631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 821.912723] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 821.921928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.932181] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 830.184647] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:35:50 (1769754950) [ 830.653854] LNet: 43724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.660226] LNetError: 43724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.665820] LNetError: 43724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 830.671415] LNet: Removed LNI 192.168.201.26@tcp [ 830.674338] LNet: Removed LNI 10.1.2.3@tcp [ 830.687332] LNet: Removed LNI 192.168.201.26@tcp2 [ 831.215545] Key type .llcrypt unregistered [ 831.221421] Key type ._llcrypt unregistered [ 838.340759] Key type ._llcrypt registered [ 838.345941] Key type .llcrypt registered [ 838.677163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 838.721058] alg: No test for adler32 (adler32-zlib) [ 839.714496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.763158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.780717] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 840.709321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.767645] LNet: 44115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.782580] LNetError: 44115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.805956] LNet: Removed LNI 192.168.201.26@tcp [ 840.852842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.973903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 841.023101] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 841.847838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 841.899693] LNet: 44145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 841.910950] LNetError: 44145:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 841.962645] LNet: Removed LNI 192.168.201.26@tcp [ 841.994896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.100107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 842.138954] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 842.521798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.541700] LNet: Removed LNI 192.168.201.26@tcp [ 842.585955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.662528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 842.707391] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 843.166292] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.341763] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.348611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.353721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.778980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.810175] LNet: 44255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.819348] LNet: 44255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 845.834137] LNetError: 44255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.842145] LNetError: 44255:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 845.860305] LNet: Removed LNI 192.168.201.26@tcp [ 845.932326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.013329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 846.044388] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 853.068567] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:36:13 (1769754973) [ 853.293917] LNet: 44848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.307846] LNetError: 44848:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.334454] LNet: Removed LNI 192.168.201.26@tcp [ 854.016473] Key type .llcrypt unregistered [ 854.024139] Key type ._llcrypt unregistered [ 862.150256] Key type ._llcrypt registered [ 862.151859] Key type .llcrypt registered [ 862.448332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.474019] alg: No test for adler32 (adler32-zlib) [ 863.471734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.549460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.586152] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 864.387817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.412252] LNet: 45248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.418668] LNetError: 45248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.432275] LNet: Removed LNI 192.168.201.26@tcp [ 864.451946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.541149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 864.577214] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 873.108397] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:36:33 (1769754993) [ 873.211186] LNet: 45831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.219214] LNetError: 45831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.258233] LNet: Removed LNI 192.168.201.26@tcp [ 874.157785] Key type .llcrypt unregistered [ 874.161274] Key type ._llcrypt unregistered [ 881.184779] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 881.341389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 881.347953] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 881.361441] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 884.061441] Key type ._llcrypt registered [ 884.063650] Key type .llcrypt registered [ 884.235520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 884.259479] alg: No test for adler32 (adler32-zlib) [ 885.122685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.150156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 885.161196] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 885.984295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.030665] LNet: 46283:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.042988] LNetError: 46283:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.086384] LNet: Removed LNI 192.168.201.26@tcp [ 886.146060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.279795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 886.351150] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 894.676744] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:36:54 (1769755014) [ 894.835866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.874121] LNet: 46877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.890105] LNetError: 46877:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.928897] LNet: Removed LNI 192.168.201.26@tcp [ 894.978818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.115344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 902.817780] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:37:03 (1769755023) [ 903.215203] LNet: 47466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.132753] Key type .llcrypt unregistered [ 904.141379] Key type ._llcrypt unregistered [ 911.173948] Key type ._llcrypt registered [ 911.178713] Key type .llcrypt registered [ 911.478752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.504774] alg: No test for adler32 (adler32-zlib) [ 912.491567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.586489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.635213] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 913.783625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.805095] LNet: 47860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 913.821082] LNetError: 47860:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 913.838648] LNet: Removed LNI 192.168.201.26@tcp [ 913.870592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.008525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 914.049947] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 921.663711] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:37:21 (1769755041) [ 921.836942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.864992] LNet: 48453:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.887305] LNetError: 48453:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.945154] LNet: Removed LNI 192.168.201.26@tcp [ 921.977766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.037315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 928.363936] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:37:28 (1769755048) [ 930.094205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 931.875276] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:37:31 (1769755051) [ 933.635986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 936.959898] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:37:35 (1769755055) [ 938.768035] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 941.546617] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:37:40 (1769755060) [ 943.241502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 946.263485] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:37:45 (1769755065) [ 949.086990] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 952.542326] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:37:51 (1769755071) [ 952.727153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 952.800460] LNet: 50500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.844914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.051112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.106542] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 954.158223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.26 [ 954.173098] LNet: 50641:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 954.178496] LNet: Removed LNI 192.168.201.26@tcp [ 962.797150] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:38:03 (1769755083) [ 963.188839] LNet: 51212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.202896] LNetError: 51212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.236543] LNet: Removed LNI 192.168.201.26@tcp [ 964.237667] Key type .llcrypt unregistered [ 964.250715] Key type ._llcrypt unregistered [ 972.603983] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 972.816536] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 972.821394] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 972.825191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 975.570959] Key type ._llcrypt registered [ 975.576331] Key type .llcrypt registered [ 975.813877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.841307] alg: No test for adler32 (adler32-zlib) [ 977.203441] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 984.109712] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:38:24 (1769755104) [ 984.309223] LNet: 52105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.322891] LNetError: 52105:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 984.348360] LNet: Removed LNI 10.1.2.3@tcp [ 985.230868] Key type .llcrypt unregistered [ 985.234460] Key type ._llcrypt unregistered [ 991.442804] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 991.833213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 991.836479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 991.838979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 994.681945] Key type ._llcrypt registered [ 994.683770] Key type .llcrypt registered [ 994.913977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 994.936563] alg: No test for adler32 (adler32-zlib) [ 995.855817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1002.729554] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:38:42 (1769755122) [ 1002.914870] LNet: 52997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.924806] LNetError: 52997:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.948948] LNet: Removed LNI 10.1.2.3@tcp [ 1003.589336] Key type .llcrypt unregistered [ 1003.591990] Key type ._llcrypt unregistered [ 1009.985402] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.380172] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.388371] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.394981] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1012.055959] Key type ._llcrypt registered [ 1012.059203] Key type .llcrypt registered [ 1012.218447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1012.237395] alg: No test for adler32 (adler32-zlib) [ 1013.167711] LNet: 53291:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1013.176839] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1020.194199] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:39:00 (1769755140) [ 1020.529537] LNet: 53883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.537168] LNetError: 53883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1020.553074] LNet: Removed LNI 10.1.2.3@tcp [ 1021.212750] Key type .llcrypt unregistered [ 1021.215261] Key type ._llcrypt unregistered [ 1027.065229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1027.253779] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1027.256830] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1027.258972] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1029.940631] Key type ._llcrypt registered [ 1029.943037] Key type .llcrypt registered [ 1030.151738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.170326] alg: No test for adler32 (adler32-zlib) [ 1031.140525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.198964] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1040.865763] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:39:20 (1769755160) [ 1041.383558] LNet: 55110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.389543] LNetError: 55110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.398637] LNet: Removed LNI 10.1.2.3@tcp [ 1042.091043] Key type .llcrypt unregistered [ 1042.093242] Key type ._llcrypt unregistered [ 1048.639545] Key type ._llcrypt registered [ 1048.642570] Key type .llcrypt registered [ 1048.816065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1048.829686] alg: No test for adler32 (adler32-zlib) [ 1049.856679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1049.903930] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1059.578472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1060.539030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_interrupt -r 1 [ 1060.594292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_interrupt -r 1 [ 1060.624815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1060.643803] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1060.844708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1060.871311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1060.902996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.932916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.959912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.986633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.010684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.035479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.054903] LNet: 55545:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.070329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.097870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.125287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.154361] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1061.159631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.163136] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1061.194611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.219797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.250669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1061.753170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_dropped -r 1 [ 1061.787968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_dropped -r 1 [ 1061.818964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1062.066665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.094460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1062.121464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.148195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.179587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.213593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.248080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.273252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.289270] LNet: 55654:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.307162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.337324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.363380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.402182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.440626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.475220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.508962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1063.124609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_aborted -r 1 [ 1063.150217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_aborted -r 1 [ 1063.182122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1063.200596] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1063.203715] LNet: Skipped 1 previous similar message [ 1063.390061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.418851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1063.444936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.467460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.490258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.517945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.546196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.570841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.587331] LNet: 55764:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.607182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.640232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.686860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.723292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.746526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.773494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.800915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1064.287278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_no_route -r 1 [ 1064.329607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_no_route -r 1 [ 1064.362798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1064.569202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1064.595792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1064.619294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1064.640996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1064.674625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.701188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.726627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.757816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.789554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.818269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.857487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.895194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.920239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.956937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.984148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.511075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_timeout -r 1 [ 1065.538371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_timeout -r 1 [ 1065.577914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1065.597900] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1065.601975] LNet: Skipped 1 previous similar message [ 1065.793805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.826474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1065.856618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.893130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.915584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.942223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.968490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.004811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.026633] LNet: 55982:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.032449] LNet: 55982:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1066.051617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.084308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.115284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.144164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.176484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.218771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1066.267583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1066.929357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_error -r 1 [ 1066.976561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_error -r 1 [ 1067.019309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1067.232811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1067.267559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1067.301285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1067.330313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1067.374178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1067.430145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1067.491248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1067.525314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1067.583719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1067.637062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1067.681995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.745540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1067.807052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1067.862418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.920216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1068.352056] LNet: 56132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.358625] LNetError: 56132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1068.382919] LNet: Removed LNI 192.168.201.26@tcp [ 1068.895221] Key type .llcrypt unregistered [ 1068.896409] Key type ._llcrypt unregistered [ 1086.134922] Key type ._llcrypt registered [ 1086.136223] Key type .llcrypt registered [ 1087.489468] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:40:07 (1769755207) [ 1088.326841] Key type .llcrypt unregistered [ 1088.329976] Key type ._llcrypt unregistered [ 1094.077451] Key type ._llcrypt registered [ 1094.080775] Key type .llcrypt registered [ 1094.287765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1094.310716] alg: No test for adler32 (adler32-zlib) [ 1095.220894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1095.279418] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1103.300929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1115.632150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.715480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1115.763707] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1117.291162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_interrupt -r 1 [ 1117.328220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 1 [ 1117.357379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_interrupt -r 1 [ 1117.394442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 1 [ 1117.430305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_interrupt -r 1 [ 1117.464133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 1 [ 1117.492495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_interrupt -r 1 [ 1117.523955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 1 [ 1117.563357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1117.583517] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1117.591379] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1117.812384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.849230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1117.889802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.928416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.969760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.996333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.037399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.067171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.086477] LNet: 57595:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.105839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.139524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.171613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.207575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.236813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.282286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.328215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1118.951239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_dropped -r 1 [ 1118.999631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 1 [ 1119.042450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_dropped -r 1 [ 1119.091792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 1 [ 1119.131575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_dropped -r 1 [ 1119.175700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 1 [ 1119.208276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_dropped -r 1 [ 1119.238917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 1 [ 1119.272142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1119.288358] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1119.294582] LNet: Skipped 2 previous similar messages [ 1119.521540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.549516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1119.576463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.604472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.632946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.659521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.690774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.718601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.736610] LNet: 57717:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1119.752909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.786532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.812506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.842300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.871503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.897581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.922295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1120.380870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_aborted -r 1 [ 1120.417316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 1 [ 1120.452406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_aborted -r 1 [ 1120.485957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 1 [ 1120.525654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_aborted -r 1 [ 1120.565643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 1 [ 1120.603984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_aborted -r 1 [ 1120.639390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 1 [ 1120.678919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1120.699542] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1120.705438] LNet: Skipped 2 previous similar messages [ 1121.001297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.049113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1121.083312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.110556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.161643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.195218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.230773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.284212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.309459] LNet: 57838:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.335835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.371241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.435377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.493274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.534979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.574802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.623921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.276223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_no_route -r 1 [ 1122.320398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 1 [ 1122.371689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_no_route -r 1 [ 1122.418287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 1 [ 1122.447860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_no_route -r 1 [ 1122.485166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 1 [ 1122.541644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_no_route -r 1 [ 1122.586410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 1 [ 1122.644688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1122.886595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1122.920367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1122.987362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.035469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.073963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.138819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.181676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.226311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.258947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.286780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.333192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.423463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.469632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.509355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.538577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.112813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_timeout -r 1 [ 1124.146459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 1 [ 1124.177326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_timeout -r 1 [ 1124.221607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 1 [ 1124.259264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_timeout -r 1 [ 1124.297675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 1 [ 1124.344210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_timeout -r 1 [ 1124.397201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 1 [ 1124.458085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1124.502511] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1124.507337] LNet: Skipped 5 previous similar messages [ 1124.765794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.806829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1124.850841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.895390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.930329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.979816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.029694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.090485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.116971] LNet: 58081:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.127049] LNet: 58081:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1125.160290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.189359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.233499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.284682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.331976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.356987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.383493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1125.897236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_error -r 1 [ 1125.931600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_error -r 1 [ 1125.976407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_error -r 1 [ 1126.021360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_error -r 1 [ 1126.083983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_error -r 1 [ 1126.170974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_error -r 1 [ 1126.207737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_error -r 1 [ 1126.250762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_error -r 1 [ 1126.285533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1126.526334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.544226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1126.579777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.613900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.642672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.669438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.705940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.752409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.794322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.831964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.863800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.899828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.933734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.965422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.987707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1132.568918] LNet: 58277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.581962] LNetError: 58277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.593660] LNet: Removed LNI 192.168.201.26@tcp [ 1132.635919] LNet: Removed LNI 192.168.201.26@tcp1 [ 1133.207871] Key type .llcrypt unregistered [ 1133.214749] Key type ._llcrypt unregistered [ 1150.405766] Key type ._llcrypt registered [ 1150.407397] Key type .llcrypt registered [ 1151.858984] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:41:11 (1769755271) [ 1153.169330] Key type .llcrypt unregistered [ 1153.170869] Key type ._llcrypt unregistered [ 1159.766425] Key type ._llcrypt registered [ 1159.768519] Key type .llcrypt registered [ 1160.099773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.132421] alg: No test for adler32 (adler32-zlib) [ 1161.106779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.149132] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1168.544900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1169.601588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_dropped -r 1 [ 1169.650959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_dropped -r 1 [ 1169.700528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1169.731584] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1170.048517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.082220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1170.132442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.165275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.201778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.266705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.323634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1170.367386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1170.399740] LNet: 59522:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1170.433548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1170.469397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1170.500293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1170.543427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1170.581472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.611806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.647745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.184272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_error -r 1 [ 1171.221885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_error -r 1 [ 1171.258893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1171.281973] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1171.658232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.698845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1171.772137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.849354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.918249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.967034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1172.008331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1172.054155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1172.077770] LNet: 59631:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1172.110893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1172.153589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1172.205496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.240519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1172.278960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1172.312561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.349988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.905952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_timeout -r 1 [ 1172.942665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_timeout -r 1 [ 1172.996839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1173.036303] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1173.394730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.447395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1173.512093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.585685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.646243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.717316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.769518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.822732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.839911] LNet: 59740:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.862797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.916896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.978518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.049843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.121719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.192452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.249499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.528920] LNet: 59780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1174.539109] LNetError: 59780:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1175.589975] LNet: Removed LNI 192.168.201.26@tcp [ 1176.123573] Key type .llcrypt unregistered [ 1176.125335] Key type ._llcrypt unregistered [ 1196.299656] Key type ._llcrypt registered [ 1196.309132] Key type .llcrypt registered [ 1198.834342] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:41:57 (1769755317) [ 1200.076645] Key type .llcrypt unregistered [ 1200.078675] Key type ._llcrypt unregistered [ 1209.305344] Key type ._llcrypt registered [ 1209.307091] Key type .llcrypt registered [ 1209.545228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1209.557769] alg: No test for adler32 (adler32-zlib) [ 1210.522445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1210.542412] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1218.053288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1227.197685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1227.270661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1227.315397] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1228.829768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_dropped -r 1 [ 1228.864432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 1 [ 1228.904660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_dropped -r 1 [ 1228.959517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 1 [ 1228.999571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_dropped -r 1 [ 1229.040664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 1 [ 1229.073705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_dropped -r 1 [ 1229.120619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 1 [ 1229.172540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1229.206085] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1229.209407] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 1229.485863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1229.537796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1229.597725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1229.658993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1229.718741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1229.778758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1229.823469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1229.872958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1229.895098] LNet: 61246:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1229.922178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1229.966620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.012721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.055211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.106752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.169160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1230.239356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.310538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.960167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_error -r 1 [ 1230.990945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_error -r 1 [ 1231.019950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_error -r 1 [ 1231.051314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_error -r 1 [ 1231.098518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_error -r 1 [ 1231.144499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_error -r 1 [ 1231.199527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_error -r 1 [ 1231.233234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_error -r 1 [ 1231.287317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1231.318527] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1231.325885] LNet: Skipped 2 previous similar messages [ 1231.602353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1231.633692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1231.675314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1231.725561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1231.774714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1231.817378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1231.842656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1231.858345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1231.876500] LNet: 61370:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1231.895414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1231.952778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1231.997195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1232.043573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1232.123709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1232.192438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1232.300200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1232.369438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.212922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_timeout -r 1 [ 1233.274341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_timeout -r 1 [ 1233.342329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_timeout -r 1 [ 1233.424180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_timeout -r 1 [ 1233.502403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_timeout -r 1 [ 1233.541383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_timeout -r 1 [ 1233.579541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_timeout -r 1 [ 1233.622541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_timeout -r 1 [ 1233.656834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.126@tcp [ 1233.677511] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1233.943542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1233.986244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1234.018278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.049245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.087376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.121692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.159293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.193919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.211691] LNet: 61493:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.229476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.257626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.302186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.332776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.364180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.391352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.419213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1234.448328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.728988] LNet: 61570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1239.737862] LNetError: 61570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1239.744184] LNet: Removed LNI 192.168.201.26@tcp [ 1239.764583] LNet: Removed LNI 192.168.201.26@tcp1 [ 1240.780688] Key type .llcrypt unregistered [ 1240.790050] Key type ._llcrypt unregistered [ 1266.306958] Key type ._llcrypt registered [ 1266.309939] Key type .llcrypt registered [ 1268.264838] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:43:07 (1769755387) [ 1268.869461] Key type .llcrypt unregistered [ 1268.872137] Key type ._llcrypt unregistered [ 1275.113300] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1275.299161] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1275.303556] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1275.306725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1277.945746] Key type ._llcrypt registered [ 1277.947841] Key type .llcrypt registered [ 1278.222743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.253165] alg: No test for adler32 (adler32-zlib) [ 1279.316889] LNet: 62637:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1279.351200] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1279.439983] LNet: 62651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.446337] LNetError: 62651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.471466] LNet: Removed LNI 192.168.201.26@tcp [ 1280.488340] Key type .llcrypt unregistered [ 1280.495319] Key type ._llcrypt unregistered [ 1288.474510] Key type ._llcrypt registered [ 1288.476862] Key type .llcrypt registered [ 1288.623696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1288.667109] alg: No test for adler32 (adler32-zlib) [ 1289.532985] LNet: 62907:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1289.541115] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1289.543056] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1289.574695] LNet: 62921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1289.578285] LNetError: 62921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.583242] LNet: Removed LNI 192.168.201.26@tcp [ 1289.592649] LNet: Removed LNI 10.1.2.3@tcp1 [ 1290.139554] Key type .llcrypt unregistered [ 1290.142664] Key type ._llcrypt unregistered [ 1298.611479] Key type ._llcrypt registered [ 1298.613903] Key type .llcrypt registered [ 1298.864047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1298.890307] alg: No test for adler32 (adler32-zlib) [ 1299.912428] LNet: 63176:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1299.943179] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1299.947258] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1300.041259] LNet: 63190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1300.062509] LNetError: 63190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1300.078925] LNet: Removed LNI 192.168.201.26@tcp [ 1300.123307] LNet: Removed LNI 10.1.2.3@tcp [ 1301.087702] Key type .llcrypt unregistered [ 1301.091247] Key type ._llcrypt unregistered [ 1307.594639] Key type ._llcrypt registered [ 1307.600594] Key type .llcrypt registered [ 1307.993907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1308.069138] alg: No test for adler32 (adler32-zlib) [ 1309.142255] LNet: 63447:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1309.194298] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1309.207981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1309.285132] LNet: 63461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1309.298298] LNetError: 63461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.305876] LNet: Removed LNI 192.168.201.26@tcp [ 1309.337354] LNet: Removed LNI 10.1.2.3@tcp [ 1310.578948] Key type .llcrypt unregistered [ 1310.583494] Key type ._llcrypt unregistered [ 1318.267588] Key type ._llcrypt registered [ 1318.270154] Key type .llcrypt registered [ 1318.535748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1318.559951] alg: No test for adler32 (adler32-zlib) [ 1319.477857] LNet: 63721:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1319.484646] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1319.486205] LNetError: 63721:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1319.490386] LNetError: Error -100 starting up LNI tcp [ 1319.499225] LNet: Removed LNI 192.168.201.26@tcp [ 1320.054627] Key type .llcrypt unregistered [ 1320.062552] Key type ._llcrypt unregistered [ 1334.564609] Key type ._llcrypt registered [ 1334.566923] Key type .llcrypt registered [ 1336.027699] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:44:16 (1769755456) [ 1336.889403] Key type .llcrypt unregistered [ 1336.892110] Key type ._llcrypt unregistered [ 1342.755831] Key type ._llcrypt registered [ 1342.758801] Key type .llcrypt registered [ 1342.965869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1342.987426] alg: No test for adler32 (adler32-zlib) [ 1343.964952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1344.028740] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1353.011237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1353.895318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e network_timeout -r 1 [ 1353.963955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e network_timeout -r 1 [ 1354.005527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1354.041921] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1354.274765] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1354.347844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.387269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1354.425866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1354.463893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.498875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.556627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1354.627920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1354.667494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1354.691857] LNet: 64970:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1354.718588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.752241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.797231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.840462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.898128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.955742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.343401] LNet: 65008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.351355] LNetError: 65008:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.367670] LNet: Removed LNI 192.168.201.26@tcp [ 1355.907097] Key type .llcrypt unregistered [ 1355.914469] Key type ._llcrypt unregistered [ 1376.720459] Key type ._llcrypt registered [ 1376.723860] Key type .llcrypt registered [ 1377.069838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1377.100234] alg: No test for adler32 (adler32-zlib) [ 1378.094492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1378.156039] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1388.196601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1398.068187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.126400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1398.195791] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1399.596194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e network_timeout -r 1 [ 1399.638415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e network_timeout -r 1 [ 1399.675740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e network_timeout -r 1 [ 1399.707750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e network_timeout -r 1 [ 1399.746762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e network_timeout -r 1 [ 1399.792866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e network_timeout -r 1 [ 1399.814702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e network_timeout -r 1 [ 1399.861130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e network_timeout -r 1 [ 1399.896778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1399.909197] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1400.158808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1400.191497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1400.215538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1400.265734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1400.311574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1400.338147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1400.368590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1400.406340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1400.422620] LNet: 65882:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1400.437485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1400.465663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1400.495308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1400.532711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1400.567319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1400.596382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1405.581993] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1405.855228] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1406.044394] LNet: 65955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.056278] LNetError: 65955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.066990] LNet: Removed LNI 192.168.201.26@tcp [ 1406.081450] LNet: Removed LNI 192.168.201.26@tcp1 [ 1406.714261] Key type .llcrypt unregistered [ 1406.715765] Key type ._llcrypt unregistered [ 1428.977739] Key type ._llcrypt registered [ 1428.984790] Key type .llcrypt registered [ 1430.734385] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:45:50 (1769755550) [ 1431.235799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.254216] alg: No test for adler32 (adler32-zlib) [ 1432.130369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.163948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.188465] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1433.210763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1433.268954] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1434.327907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1434.418944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1434.469579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1434.504789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1434.543694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1434.581900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1434.631567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.26@tcp [ 1434.657683] LNetError: 67053:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1435.231559] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1435.231707] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1437.279570] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1438.246281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1441.376558] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1441.930719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1442.027994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1442.081157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.094948] LNet: 67160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1442.100966] LNetError: 67160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1442.104814] LNet: Removed LNI 192.168.201.26@tcp [ 1442.123783] LNet: Removed LNI 192.168.201.26@tcp1 [ 1442.146659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.208339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.239851] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1442.834311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1442.864719] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1444.301561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1444.350406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1444.400426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1444.438836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1444.503727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1444.583717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1444.655525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1444.689557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.26@tcp [ 1444.726880] LNetError: 67432:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1445.279858] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1448.446462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1451.423942] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1451.435432] LNet: Skipped 1 previous similar message [ 1456.528768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1456.569419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1456.609544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1464.608277] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:46:24 (1769755584) [ 1464.828928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.893444] LNet: 68150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.900443] LNetError: 68150:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.911413] LNetError: 68150:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1464.936201] LNet: Removed LNI 192.168.201.26@tcp [ 1464.953601] LNet: Removed LNI 192.168.201.26@tcp1 [ 1464.990497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.081694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1465.117943] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1465.928670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1465.974856] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1466.934201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1467.091599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1467.129374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1467.187222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1467.217210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1467.252957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1467.311952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.26@tcp --health 0 [ 1467.423511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1468.127708] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.26@tcp [ 1468.136508] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1468.148066] LNet: Skipped 1 previous similar message [ 1483.178826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1483.357933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1483.409765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.460240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.26@tcp --health 500 [ 1483.566139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1490.236327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1490.357052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1490.380136] LNet: 68642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.387668] LNetError: 68642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.394461] LNetError: 68642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1490.400991] LNet: Removed LNI 192.168.201.26@tcp [ 1490.424946] LNet: Removed LNI 192.168.201.26@tcp1 [ 1490.476122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.534990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.561744] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1491.484174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.516707] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1492.438211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1492.483942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1492.547289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1492.597401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1492.641127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1492.674644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1492.715756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1492.757540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.26@tcp --health 0 [ 1493.599750] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1493.608583] LNet: Skipped 3 previous similar messages [ 1494.182807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.844907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1504.903743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1504.978517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1513.918343] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:47:13 (1769755633) [ 1514.444320] LNet: 69689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.449707] LNetError: 69689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.454903] LNetError: 69689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1514.460624] LNet: Removed LNI 192.168.201.26@tcp [ 1514.474089] LNet: Removed LNI 192.168.201.26@tcp1 [ 1515.229140] Key type .llcrypt unregistered [ 1515.233944] Key type ._llcrypt unregistered [ 1522.482660] Key type ._llcrypt registered [ 1522.486297] Key type .llcrypt registered [ 1522.702580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.716313] alg: No test for adler32 (adler32-zlib) [ 1523.831783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1523.877969] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1532.566618] Lustre: DEBUG MARKER: Initial discovery [ 1532.615325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.126@tcp [ 1535.098868] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1535.326563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1535.399746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.126@tcp [ 1535.409586] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1535.418888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1535.436470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.126@tcp -r 1 -m GET -l 3 [ 1535.460798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.126@tcp -r 1 -m GET -e local_error [ 1535.501377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.126@tcp -r 1 -m PUT -l 6 [ 1536.163773] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1537.377116] Lustre: DEBUG MARKER: Force oleg126-client.virtnet to discover 192.168.201.126@tcp (in background) [ 1537.485386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.126@tcp [ 1538.888340] Lustre: DEBUG MARKER: Force oleg126-server to discover 192.168.201.26@tcp [ 1540.577436] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1542.253307] Lustre: DEBUG MARKER: Wait for 70442 [ 1544.486309] Lustre: DEBUG MARKER: Finished wait on 70442 [ 1544.577210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1544.635549] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1544.660689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1544.976914] LNet: 70898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1544.983694] LNetError: 70898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.003316] LNet: Removed LNI 192.168.201.26@tcp [ 1545.432696] Key type .llcrypt unregistered [ 1545.434132] Key type ._llcrypt unregistered [ 1561.788899] Key type ._llcrypt registered [ 1561.791216] Key type .llcrypt registered [ 1563.116964] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:48:03 (1769755683) [ 1563.790449] Key type .llcrypt unregistered [ 1563.792651] Key type ._llcrypt unregistered [ 1571.186282] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1571.465704] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1571.479606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1571.485848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1573.961589] Key type ._llcrypt registered [ 1573.962731] Key type .llcrypt registered [ 1574.113119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1574.125655] alg: No test for adler32 (adler32-zlib) [ 1575.044257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.088271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.111899] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1575.698774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1575.715694] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1583.556985] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:48:23 (1769755703) [ 1583.895419] LNet: 72968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.907523] LNetError: 72968:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.910762] LNet: Removed LNI 192.168.201.26@tcp [ 1583.941490] LNet: Removed LNI 10.1.2.3@tcp [ 1584.568536] Key type .llcrypt unregistered [ 1584.572300] Key type ._llcrypt unregistered [ 1591.472446] Key type ._llcrypt registered [ 1591.474767] Key type .llcrypt registered [ 1591.680477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1591.711132] alg: No test for adler32 (adler32-zlib) [ 1592.775518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1592.862890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1592.904730] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1593.745686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1593.769100] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1594.823113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.26@tcp [ 1595.871321] LNetError: Unexpected error -22 connecting to 192.168.201.26@tcp at host 192.168.201.26:988 [ 1602.023926] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:48:42 (1769755722) [ 1602.281299] LNet: 74094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.297900] LNetError: 74094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1602.316971] LNet: Removed LNI 192.168.201.26@tcp [ 1602.356155] LNet: Removed LNI 10.1.2.3@tcp [ 1602.863308] Key type .llcrypt unregistered [ 1602.867648] Key type ._llcrypt unregistered [ 1609.446717] Key type ._llcrypt registered [ 1609.449731] Key type .llcrypt registered [ 1609.801238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1609.826365] alg: No test for adler32 (adler32-zlib) [ 1610.826714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.901737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.930558] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1612.101472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1612.140072] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 1613.283791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.26@tcp1 --nid 192.168.201.26@tcp2 [ 1618.305956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp1 192.168.201.26@tcp2 [ 1618.326729] LNetError: 74927:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.26@tcp1 to 192.168.201.26@tcp2 - Route table may be misconfigured [ 1618.347285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp2 192.168.201.26@tcp1 [ 1625.890312] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:49:06 (1769755746) [ 1626.283066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.343832] LNet: 75500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.350776] LNetError: 75500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.357959] LNet: Removed LNI 192.168.201.26@tcp1 [ 1627.433515] LNet: Removed LNI 192.168.201.26@tcp2 [ 1627.524106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.626499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1627.670450] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1628.603986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1628.630083] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1629.634848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 1629.676240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.26@tcp [ 1629.704444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp1 192.168.201.26@tcp1 [ 1629.743754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp -d 192.168.201.26@tcp -e network_timeout [ 1629.815832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -e network_timeout [ 1629.855838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -e network_timeout [ 1629.902756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -e network_timeout [ 1629.936496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1629.959502] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1629.975088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1630.134720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1630.266021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1630.625186] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1637.457117] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:49:17 (1769755757) [ 1637.571386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1637.603460] LNet: 76354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.615624] LNetError: 76354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.623713] LNetError: 76354:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1637.631851] LNet: Removed LNI 192.168.201.26@tcp [ 1638.695061] LNet: Removed LNI 192.168.201.26@tcp1 [ 1638.734607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.875691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1639.205875] LNet: 76398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.591450] Key type .llcrypt unregistered [ 1639.595302] Key type ._llcrypt unregistered [ 1651.291918] Key type ._llcrypt registered [ 1651.294440] Key type .llcrypt registered [ 1652.713465] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:49:32 (1769755772) [ 1653.422374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.470973] alg: No test for adler32 (adler32-zlib) [ 1654.432706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.819344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1657.728082] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:49:37 (1769755777) [ 1657.890298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.905149] LNet: 77495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.937829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.988906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.010691] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1658.694610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1658.722959] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1659.594233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1659.642392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp1 [ 1659.686686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp1 [ 1666.709625] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:49:46 (1769755786) [ 1668.573525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg126-server [ 1670.483849] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:49:50 (1769755790) [ 1672.474900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg126-server [ 1674.210569] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:49:54 (1769755794) [ 1676.388984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg126-server [ 1678.269605] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:49:58 (1769755798) [ 1680.182415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg126-server [ 1681.599203] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:50:02 (1769755802) [ 1683.834184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg126-server [ 1686.021837] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:50:05 (1769755805) [ 1688.046602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg126-server [ 1690.019272] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:50:10 (1769755810) [ 1692.224502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg126-server [ 1694.095791] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:50:14 (1769755814) [ 1696.045093] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg126-server [ 1697.718176] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:50:18 (1769755818) [ 1699.422370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg126-server [ 1701.399297] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:50:21 (1769755821) [ 1701.738536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.756811] LNet: 81372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.765157] LNetError: 81372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.771516] LNet: Removed LNI 192.168.201.26@tcp [ 1702.821707] LNet: Removed LNI 192.168.201.26@tcp1 [ 1702.882472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.963056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.014526] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1703.719535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1703.781873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1703.969445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.981972] LNet: 81519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.986975] LNetError: 81519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.995411] LNetError: 81519:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1704.015357] LNet: Removed LNI 192.168.201.26@tcp [ 1704.039645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.081621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1704.094609] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1704.828460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1704.996357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.019937] LNetError: 81664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.087893] LNet: Removed LNI 192.168.201.26@tcp [ 1706.125771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.196283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.222370] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1707.020905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1707.129626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1708.491828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.518991] LNet: 81821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.525525] LNet: 81821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1708.535430] LNetError: 81821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.608513] LNet: Removed LNI 192.168.201.26@tcp [ 1709.655230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.739144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1709.776795] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1710.699919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1710.984561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.044964] LNet: Removed LNI 192.168.201.26@tcp [ 1712.137957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.258618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1712.302922] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1713.209617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1713.308915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1713.620613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.670865] LNet: 82114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.679364] LNet: 82114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1713.696724] LNetError: 82114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.704833] LNetError: 82114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1714.725277] LNet: Removed LNI 192.168.201.26@tcp [ 1714.771514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.892454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1714.924055] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1716.036220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1716.269394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.355738] LNet: Removed LNI 192.168.201.26@tcp [ 1717.418424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.519622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1717.555387] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1718.362317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1718.467600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1718.771279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.844848] LNet: Removed LNI 192.168.201.26@tcp [ 1719.919779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.003795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1720.054599] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1720.924806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1721.123697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.162415] LNet: Removed LNI 192.168.201.26@tcp [ 1722.226773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.329597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1722.385389] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1723.218888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1723.292500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1723.508695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.535735] LNet: 82700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.540603] LNet: 82700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1723.547707] LNetError: 82700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.553798] LNetError: 82700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1723.571795] LNet: Removed LNI 192.168.201.26@tcp [ 1723.601661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.667773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1723.700085] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1724.468786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1725.873048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.955408] LNet: Removed LNI 192.168.201.26@tcp [ 1727.000812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.112597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.151086] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1727.920039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1727.998771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1728.258483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.321153] LNet: Removed LNI 192.168.201.26@tcp [ 1729.356068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.431314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1729.477453] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1730.335504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 1730.662651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.761894] LNet: Removed LNI 192.168.201.26@tcp [ 1731.843225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.919889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1731.956737] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1733.119915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1733.217746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.255299] LNet: Removed LNI 192.168.201.26@tcp [ 1733.284525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.395625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.470052] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1734.234104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1741.779288] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:51:01 (1769755861) [ 1742.165315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1742.182846] LNet: 83985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1742.185799] LNet: 83985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1742.203292] LNetError: 83985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1742.208247] LNetError: 83985:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1742.231172] LNet: Removed LNI 192.168.201.26@tcp [ 1742.255085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1742.354838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1742.401955] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1743.788803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.810921] LNet: Removed LNI 192.168.201.26@tcp231 [ 1743.826445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.888842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1743.908564] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1743.912736] LNet: 84132:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1744.391811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1744.449839] LNet: Removed LNI 192.168.201.26@tcp231 [ 1744.485828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1744.528592] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1745.788555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1745.825411] LNet: Removed LNI 192.168.201.26@tcp231 [ 1745.847507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.992481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1746.045579] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/0] [ 1746.058523] LNet: 84284:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.706120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1746.737491] LNet: Removed LNI 192.168.201.26@tcp231 [ 1746.780104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1746.832743] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/0] [ 1748.323409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.388319] LNet: Removed LNI 192.168.201.26@tcp231 [ 1748.442410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.536245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1748.586270] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/60] [ 1748.589888] LNet: 84436:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.076548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1749.108569] LNet: Removed LNI 192.168.201.26@tcp231 [ 1749.131799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1749.175057] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/60] [ 1750.328558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.384719] LNet: Removed LNI 192.168.201.26@tcp231 [ 1750.420868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.570358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1750.595751] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1750.604285] LNet: 84587:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1751.113670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1751.161350] LNet: Removed LNI 192.168.201.26@tcp231 [ 1751.187417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1751.234939] LNet: Added LNI 192.168.201.26@tcp231 [8/256/0/180] [ 1760.332270] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:51:20 (1769755880) [ 1760.457194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.519344] LNet: Removed LNI 192.168.201.26@tcp231 [ 1760.577934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.669972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1760.692364] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1761.598914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1761.638032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1761.671465] LNet: Removed LNI 192.168.201.26@tcp232 [ 1761.690320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1761.714712] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1762.498148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1762.562838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1762.605674] LNet: Removed LNI 192.168.201.26@tcp232 [ 1762.628848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1762.673327] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1763.799490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1763.828976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1763.885629] LNet: Removed LNI 192.168.201.26@tcp232 [ 1763.908192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1763.937619] LNet: Added LNI 192.168.201.26@tcp232 [8/256/0/180] [ 1765.011363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1765.078613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1765.110218] LNet: Removed LNI 192.168.201.26@tcp232 [ 1765.126264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1771.130246] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:51:31 (1769755891) [ 1772.174571] Key type .llcrypt unregistered [ 1772.176967] Key type ._llcrypt unregistered [ 1779.171380] Key type ._llcrypt registered [ 1779.172903] Key type .llcrypt registered [ 1779.363519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1779.377783] alg: No test for adler32 (adler32-zlib) [ 1780.312621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1780.334897] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1787.916372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1796.019429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.080760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1796.113405] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1797.615454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_interrupt -r 2 [ 1797.648785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 2 [ 1797.699541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_interrupt -r 2 [ 1797.740280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 2 [ 1797.767680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_interrupt -r 2 [ 1797.808710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_interrupt -r 2 [ 1797.868636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_interrupt -r 2 [ 1797.911897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_interrupt -r 2 [ 1797.938613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1797.954562] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1797.962444] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1798.178905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1798.211371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1798.244858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1798.270276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1798.314308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1798.355725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1798.412409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1798.449279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1798.472802] LNet: 87085:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1798.494974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1798.511845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1798.536896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1798.598399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1798.640345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1798.679435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1799.337267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_dropped -r 2 [ 1799.379586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 2 [ 1799.410852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_dropped -r 2 [ 1799.465287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 2 [ 1799.522405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_dropped -r 2 [ 1799.581231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_dropped -r 2 [ 1799.647228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_dropped -r 2 [ 1799.680374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_dropped -r 2 [ 1799.733512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1799.766917] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1800.074144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1800.106508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1800.166835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1800.199354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1800.234493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1800.273394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1800.304586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1800.348595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1800.378139] LNet: 87206:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1800.405296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1800.451146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1800.485828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1800.514247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1800.554592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1800.605422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1801.094714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_aborted -r 2 [ 1801.122390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 2 [ 1801.149647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_aborted -r 2 [ 1801.178294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 2 [ 1801.203298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_aborted -r 2 [ 1801.226369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_aborted -r 2 [ 1801.258557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_aborted -r 2 [ 1801.285606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_aborted -r 2 [ 1801.311813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1801.331496] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1801.560620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1801.590904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1801.628564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1801.655228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1801.683797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1801.739333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1801.759851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1801.781800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1801.793849] LNet: 87326:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1801.808775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1801.832764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1801.861517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1801.891673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1801.919922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1801.953617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1802.473907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_no_route -r 2 [ 1802.532759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 2 [ 1802.584525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_no_route -r 2 [ 1802.622917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 2 [ 1802.668541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_no_route -r 2 [ 1802.702837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_no_route -r 2 [ 1802.729922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_no_route -r 2 [ 1802.763742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_no_route -r 2 [ 1802.799723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1803.041774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1803.077622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1803.112901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1803.165948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1803.202398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1803.242974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1803.277187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1803.319832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1803.376298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1803.412300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1803.440836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1803.477306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1803.539154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1803.570835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1804.077360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e local_timeout -r 2 [ 1804.105758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 2 [ 1804.134229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e local_timeout -r 2 [ 1804.187512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 2 [ 1804.253618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e local_timeout -r 2 [ 1804.308739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e local_timeout -r 2 [ 1804.354630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e local_timeout -r 2 [ 1804.391584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e local_timeout -r 2 [ 1804.426629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1804.446225] LNet: There was an unexpected network error while writing to 192.168.201.126: rc = -22 [ 1804.449462] LNet: Skipped 1 previous similar message [ 1804.718427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1804.781160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1804.837443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1804.909837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1804.970614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1805.023864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1805.064654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1805.119379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1805.134107] LNet: 87567:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1805.137545] LNet: 87567:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1805.149875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1805.180194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1805.209407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1805.232720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1805.254742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1805.278803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1805.781117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp -m GET -e remote_dropped -r 2 [ 1805.840854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 2 [ 1805.894284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp -m GET -e remote_dropped -r 2 [ 1805.928124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 2 [ 1805.954868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp -m GET -e remote_dropped -r 2 [ 1805.987181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.126@tcp1 -m GET -e remote_dropped -r 2 [ 1806.013403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp -m GET -e remote_dropped -r 2 [ 1806.064231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.26@tcp1 -d 192.168.201.26@tcp1 -m GET -e remote_dropped -r 2 [ 1806.127146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.26@tcp 192.168.201.126@tcp [ 1806.154287] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 1806.377744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1806.402437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1806.428112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1806.463384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1806.498545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1806.525347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1806.562402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1806.590338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1806.614370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1806.639842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1806.663214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1806.684233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1806.713191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1806.733318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1812.246898] LNet: 87760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.256896] LNetError: 87760:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.261470] LNet: Removed LNI 192.168.201.26@tcp [ 1812.275042] LNet: Removed LNI 192.168.201.26@tcp1 [ 1812.712870] Key type .llcrypt unregistered [ 1812.714818] Key type ._llcrypt unregistered [ 1829.541879] Key type ._llcrypt registered [ 1829.544924] Key type .llcrypt registered [ 1830.618467] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:52:31 (1769755951) [ 1831.245671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.264894] alg: No test for adler32 (adler32-zlib) [ 1832.209240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.278176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1832.308266] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1833.115712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1833.140171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.26@tcp [ 1833.160734] LNetError: 88735:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1833.249230] LNet: 1 local NIs in recovery (showing 1): 192.168.201.26@tcp [ 1833.268928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.26@tcp [ 1833.407692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1840.121688] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:52:40 (1769755960) [ 1840.342027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.376455] LNet: 89324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.387893] LNetError: 89324:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.429147] LNet: Removed LNI 192.168.201.26@tcp [ 1840.478371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.548663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1840.574528] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1841.346284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1841.364622] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 1842.222120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.26@tcp [ 1842.239632] LNetError: 89591:0:(api-ni.c:4351:lnet_ni_set_healthv()) manually adding local NI 192.168.201.26@tcp to recovery [ 1842.599625] LNet: There was an unexpected network error while writing to 192.168.201.26: rc = -22 [ 1843.484677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.26@tcp [ 1851.132259] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:52:51 (1769755971) [ 1852.944883] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg126-server [ 1854.963495] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:52:54 (1769755974) [ 1855.079381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1855.137190] LNet: 90532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.154449] LNetError: 90532:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.168173] LNet: Removed LNI 192.168.201.26@tcp [ 1855.187403] LNet: Removed LNI 192.168.201.26@tcp1 [ 1855.258211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1855.323854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1855.356910] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1856.226382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1856.287560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1856.326599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.401447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1856.437541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.485844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1864.984879] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:53:04 (1769755984) [ 1865.153923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.205340] LNet: 91257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1865.212976] LNetError: 91257:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.217429] LNetError: 91257:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1865.252643] LNet: Removed LNI 192.168.201.26@tcp [ 1865.380180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.463874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.500507] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1873.292763] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:53:13 (1769755993) [ 1873.399761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1873.426448] LNet: 91959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1873.438858] LNetError: 91959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1873.457289] LNet: Removed LNI 192.168.201.26@tcp [ 1873.482913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1873.565374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1873.595234] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1882.885988] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:53:22 (1769756002) [ 1885.156524] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1887.255824] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:53:27 (1769756007) [ 1887.967067] LNet: 93003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.977690] LNetError: 93003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1888.005898] LNet: Removed LNI 192.168.201.26@tcp [ 1888.764852] Key type .llcrypt unregistered [ 1888.768441] Key type ._llcrypt unregistered [ 1896.575754] Key type ._llcrypt registered [ 1896.585622] Key type .llcrypt registered [ 1896.961464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1896.981815] alg: No test for adler32 (adler32-zlib) [ 1898.047761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1898.087114] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1905.910870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 1911.319959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.126@tcp [ 1911.917068] LNet: 93416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.934483] LNetError: 93416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.963055] LNet: Removed LNI 192.168.201.26@tcp [ 1912.596577] Key type .llcrypt unregistered [ 1912.606650] Key type ._llcrypt unregistered [ 1925.889666] Key type ._llcrypt registered [ 1925.892740] Key type .llcrypt registered [ 1927.447149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1929.207037] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1931.150525] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:54:11 (1769756051) [ 1931.688869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1931.699217] alg: No test for adler32 (adler32-zlib) [ 1932.723409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.867732] LNet: 94502:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.494564] Key type .llcrypt unregistered [ 1933.498139] Key type ._llcrypt unregistered [ 1941.415972] Key type ._llcrypt registered [ 1941.422668] Key type .llcrypt registered [ 1941.703977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1941.721745] alg: No test for adler32 (adler32-zlib) [ 1942.736386] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1942.739913] LNet: 94755:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1942.747669] LNet: 94755:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1942.811886] LNetError: 94767:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 1942.834537] LNetError: 94767:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.201.28@tcp has gone from up to down [ 1943.645625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.684473] LNet: 94813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1943.717481] LNetError: 94813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1943.746941] LNet: Removed LNI 192.168.201.26@tcp [ 1943.806923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.873452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1943.898418] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 1943.908645] LNet: 94820:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1943.920742] LNet: 94820:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1943.937226] LNetError: 94820:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.27@tcp is a gateway. Can not delete it [ 1943.945202] LNetError: 94820:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.201.27@tcp is a router. Can not be deleted [ 1943.962089] LNetError: 94817:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.29@tcp is being used as a gateway but routing feature is not turned on [ 1943.979534] LNetError: 94817:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 1943.988696] LNetError: 94817:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.201.29@tcp has gone from up to down [ 1944.003940] LNetError: 94817:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1944.034074] LNet: 94820:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1951.392659] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:54:31 (1769756071) [ 1954.125640] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg126-server [ 1956.693686] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:54:36 (1769756076) [ 1958.790645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg126-server [ 1960.916956] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:54:41 (1769756081) [ 1962.604097] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 1964.439171] Key type lgssc registered [ 1966.224419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1990.481264] Key type lgssc unregistered [ 1990.968234] LNet: 97241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.978554] LNetError: 97241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.042688] LNet: Removed LNI 192.168.201.26@tcp [ 1992.951617] Key type .llcrypt unregistered [ 1992.955458] Key type ._llcrypt unregistered [ 2009.343671] Key type ._llcrypt registered [ 2009.346361] Key type .llcrypt registered [ 2010.697396] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:55:30 (1769756130) [ 2011.250920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2011.273456] alg: No test for adler32 (adler32-zlib) [ 2012.236282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.331848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2012.394541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2012.466709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2012.546240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2019.541965] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:55:39 (1769756139) [ 2019.880255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.919378] LNet: 98646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.962651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.054127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2020.094189] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2021.254729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2021.268710] LNet: 98790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2021.274810] LNetError: 98790:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2021.293562] LNet: Removed LNI 192.168.201.26@tcp [ 2021.313867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.368699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2021.391352] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2022.158951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.215078] LNetError: 98818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.260954] LNet: Removed LNI 192.168.201.26@tcp [ 2022.305709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.379888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2022.404259] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2030.327847] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:55:50 (1769756150) [ 2030.728417] LNet: 99409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.735835] LNet: 99409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2030.750497] LNetError: 99409:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.775466] LNet: Removed LNI 192.168.201.26@tcp [ 2031.641226] Key type .llcrypt unregistered [ 2031.644412] Key type ._llcrypt unregistered [ 2040.034717] Key type ._llcrypt registered [ 2040.040601] Key type .llcrypt registered [ 2040.361624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2040.388815] alg: No test for adler32 (adler32-zlib) [ 2041.336563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.456340] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2041.460538] LNet: *** cfs_fail_loc=e003, val=100*** [ 2042.361668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 2042.392982] LNet: *** cfs_fail_loc=e003, val=100*** [ 2042.399108] LNet: Skipped 2 previous similar messages [ 2042.406961] Ping reply received with mapping: [ 2042.413192] found metadata reply NI[192.168.201.26@tcp] [ 2042.416288] 1a c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.419596] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.423266] 5a 5a 5a 5a 5a 5a 5a 5a [ 2042.554470] LNet: 99803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2042.558732] LNetError: 99803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2042.571300] LNet: Removed LNI 192.168.201.26@tcp [ 2043.403630] Key type .llcrypt unregistered [ 2043.406095] Key type ._llcrypt unregistered [ 2056.074756] Key type ._llcrypt registered [ 2056.077246] Key type .llcrypt registered [ 2057.666848] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:56:17 (1769756177) [ 2058.350627] Key type .llcrypt unregistered [ 2058.352354] Key type ._llcrypt unregistered [ 2065.327039] Key type ._llcrypt registered [ 2065.333862] Key type .llcrypt registered [ 2065.607633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2065.648958] alg: No test for adler32 (adler32-zlib) [ 2066.916432] LNetError: 100840:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2067.448472] Key type .llcrypt unregistered [ 2067.449953] Key type ._llcrypt unregistered [ 2073.674986] Key type ._llcrypt registered [ 2073.677510] Key type .llcrypt registered [ 2075.260285] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:56:35 (1769756195) [ 2075.829786] Key type .llcrypt unregistered [ 2075.831576] Key type ._llcrypt unregistered [ 2082.305458] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2082.323571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2085.066547] Key type ._llcrypt registered [ 2085.070952] Key type .llcrypt registered [ 2085.323460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2085.340646] alg: No test for adler32 (adler32-zlib) [ 2086.249629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2086.313679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2098.450593] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:56:57 (1769756217) [ 2098.556688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.603770] LNet: 102607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.614776] LNetError: 102607:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2098.651680] LNet: Removed LNI 10.1.2.3@tcp [ 2098.717805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.789080] Key type .llcrypt unregistered [ 2099.795749] Key type ._llcrypt unregistered [ 2107.124907] Key type ._llcrypt registered [ 2107.129815] Key type .llcrypt registered [ 2107.275170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2107.297981] alg: No test for adler32 (adler32-zlib) [ 2108.321643] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2108.408584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 2108.629463] LNet: 102900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.640925] LNetError: 102900:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.658612] LNet: Removed LNI 192.168.201.26@tcp [ 2109.424346] Key type .llcrypt unregistered [ 2109.426297] Key type ._llcrypt unregistered [ 2122.942829] Key type ._llcrypt registered [ 2122.948901] Key type .llcrypt registered [ 2124.462708] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:57:24 (1769756244) [ 2125.192355] Key type .llcrypt unregistered [ 2125.196158] Key type ._llcrypt unregistered [ 2131.683501] Key type ._llcrypt registered [ 2131.690170] Key type .llcrypt registered [ 2132.046984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2132.071130] alg: No test for adler32 (adler32-zlib) [ 2139.757203] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:57:39 (1769756259) [ 2140.005542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.104688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.278580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.328455] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2141.293503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.343868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2141.391882] LNet: Removed LNI 192.168.201.26@tcp [ 2141.747532] LNet: 104638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.152489] Key type .llcrypt unregistered [ 2142.154746] Key type ._llcrypt unregistered [ 2154.981468] Key type ._llcrypt registered [ 2154.985104] Key type .llcrypt registered [ 2156.501262] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:57:56 (1769756276) [ 2157.586514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2157.616799] alg: No test for adler32 (adler32-zlib) [ 2158.693472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.793575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.824522] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2160.036871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 2160.067955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.26@tcp [ 2160.094914] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2160.104294] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2160.113355] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2160, 0, 0 [ 2160.127433] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2160.135618] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2160.143652] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2160.151520] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2160, 0, 0 [ 2160.171894] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2160.177430] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2160.188387] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2160.194987] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2160, 0, 2210 [ 2160.211911] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2160.216930] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2160.229256] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2160.238692] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2160, 0, 2210 [ 2160.248380] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2169.111488] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:58:09 (1769756289) [ 2169.994518] LNet: 106177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.009205] LNetError: 106177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2170.029966] LNet: Removed LNI 192.168.201.26@tcp [ 2171.209804] Key type .llcrypt unregistered [ 2171.218681] Key type ._llcrypt unregistered [ 2179.660583] Key type ._llcrypt registered [ 2179.664319] Key type .llcrypt registered [ 2179.992608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2180.016968] alg: No test for adler32 (adler32-zlib) [ 2181.019752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2181.054494] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2190.538167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 2201.153715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.201226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2201.241676] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 2202.604980] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2202.696407] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2202.701959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2202.708956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2205.305341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2205.338260] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2217.896643] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:58:58 (1769756338) [ 2218.170490] LNet: 107697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2218.187323] LNetError: 107697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2218.214120] LNet: Removed LNI 192.168.201.26@tcp [ 2219.231251] LNet: Removed LNI 192.168.201.26@tcp1 [ 2219.250351] LNetError: 107697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2219.272139] LNetError: 107697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2219.294098] LNet: Removed LNI 10.1.2.3@tcp99 [ 2220.475474] Key type .llcrypt unregistered [ 2220.484166] Key type ._llcrypt unregistered [ 2227.523947] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2227.535766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2230.230341] Key type ._llcrypt registered [ 2230.233433] Key type .llcrypt registered [ 2230.713027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2230.739893] alg: No test for adler32 (adler32-zlib) [ 2231.847552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.948921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2232.004479] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2233.065195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2233.090379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2236.253501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2236.307087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2236.367256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2236.395175] LNetError: 108438:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2236.429274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2236.510637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2237.283228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2237.424804] LNet: 108458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.437402] LNetError: 108458:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2237.448448] LNet: Removed LNI 192.168.201.26@tcp [ 2237.461039] LNet: Removed LNI 10.1.2.3@tcp [ 2238.239852] Key type .llcrypt unregistered [ 2238.241502] Key type ._llcrypt unregistered [ 2253.610955] Key type ._llcrypt registered [ 2253.612475] Key type .llcrypt registered [ 2255.369212] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:59:35 (1769756375) [ 2255.941129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2255.966190] alg: No test for adler32 (adler32-zlib) [ 2256.953307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.002795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2257.025967] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2257.760362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.26@tcp [ 2257.903562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg126-client@tcp [ 2266.126233] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:59:46 (1769756386) [ 2266.792958] LNet: 109992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2266.816579] LNetError: 109992:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2266.864222] LNet: Removed LNI 192.168.201.26@tcp [ 2267.886565] Key type .llcrypt unregistered [ 2267.890342] Key type ._llcrypt unregistered [ 2274.142762] Key type ._llcrypt registered [ 2274.145747] Key type .llcrypt registered [ 2274.390520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2274.427755] alg: No test for adler32 (adler32-zlib) [ 2275.488995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2275.537135] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2275.599343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2285.144551] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:00:04 (1769756404) [ 2287.019550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2290.642976] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:00:09 (1769756409) [ 2293.488739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2295.914216] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:00:15 (1769756415) [ 2296.135980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.199781] LNet: 111405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.210156] LNetError: 111405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2296.233958] LNet: Removed LNI 192.168.201.26@tcp [ 2296.282609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2296.368401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2296.438138] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2297.517250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2297.559833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2305.071944] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:00:25 (1769756425) [ 2305.269711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.323631] LNet: 112119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.345786] LNetError: 112119:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2306.411455] LNet: Removed LNI 192.168.201.26@tcp [ 2306.490445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.600815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2306.669987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2307.070420] LNet: 112155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.481316] Key type .llcrypt unregistered [ 2307.482980] Key type ._llcrypt unregistered [ 2320.259444] Key type ._llcrypt registered [ 2320.261464] Key type .llcrypt registered [ 2321.749683] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:00:41 (1769756441) [ 2322.328977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2322.358764] alg: No test for adler32 (adler32-zlib) [ 2323.374362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2323.472808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2323.511644] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2324.374294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2324.491994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2333.063088] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:00:52 (1769756452) [ 2333.273344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.295520] LNet: 113668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.302145] LNetError: 113668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2334.381706] LNet: Removed LNI 192.168.201.26@tcp [ 2334.436640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.508659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2334.578364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2342.653688] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:01:03 (1769756463) [ 2342.947922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.967848] LNet: 114266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2342.992580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.062445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2343.480305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2343.527661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2343.615972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2344.326297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2344.369468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2344.517338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.562786] LNet: 114299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.584873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.672843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2345.129387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2345.169886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2345.227996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2345.614772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2345.653278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2352.895814] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:01:13 (1769756473) [ 2353.123389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.177780] LNet: 114900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.209845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.283718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.316383] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2354.262806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2354.297179] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2355.308779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2355.385880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2355.752984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.797669] LNetError: 115179:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.807709] LNet: Removed LNI 192.168.201.26@tcp [ 2355.825769] LNet: Removed LNI 192.168.201.26@tcp2 [ 2355.871789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.949966] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2355.967056] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2355.974671] LNet: 115185:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.781778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2357.036846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2357.101181] LNetError: 115222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2357.112238] LNetError: 115222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2357.115629] LNet: Removed LNI 192.168.201.26@tcp [ 2357.122952] LNet: Removed LNI 192.168.201.26@tcp2 [ 2357.145161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2357.249760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2357.287734] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2358.474281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2358.528268] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2359.399158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2359.469623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2359.521995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2359.754494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2359.796540] LNet: 115507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2359.804319] LNet: 115507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2359.816888] LNetError: 115507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2359.826674] LNetError: 115507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2359.834648] LNet: Removed LNI 192.168.201.26@tcp [ 2360.885366] LNet: Removed LNI 192.168.201.26@tcp2 [ 2360.929121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2361.037676] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2361.044723] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2361.051134] LNet: 115513:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.853964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2362.156790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2362.201971] LNet: Removed LNI 192.168.201.26@tcp [ 2363.240916] LNet: Removed LNI 192.168.201.26@tcp2 [ 2363.338674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.487573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2363.532752] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2364.364676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2364.387465] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2365.602629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2365.646199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2365.905494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.927821] LNetError: 115830:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.930637] LNetError: 115830:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2365.934950] LNet: Removed LNI 192.168.201.26@tcp [ 2365.954473] LNet: Removed LNI 192.168.201.26@tcp2 [ 2365.974750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2366.055959] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2366.063220] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2366.070655] LNet: 115836:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.970227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2367.187626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2368.223173] LNet: Removed LNI 192.168.201.26@tcp [ 2368.245110] LNet: Removed LNI 192.168.201.26@tcp2 [ 2368.316144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.443757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2368.478107] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2369.723559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2369.743511] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2370.989369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2371.035666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2371.114659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2371.392032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.421524] LNet: 116157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.432784] LNet: 116157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2372.511162] LNet: Removed LNI 192.168.201.26@tcp [ 2373.543180] LNet: Removed LNI 192.168.201.26@tcp2 [ 2373.598597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.738555] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2373.750054] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2373.761746] LNet: 116164:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.405817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2374.610738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.636634] LNetError: 116201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2374.640922] LNetError: 116201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2374.646220] LNet: Removed LNI 192.168.201.26@tcp [ 2375.742192] LNet: Removed LNI 192.168.201.26@tcp2 [ 2375.838947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2375.955251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2375.999965] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2376.961275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2376.979901] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2377.749923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2377.793815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2377.870805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2378.226231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.295229] LNet: Removed LNI 192.168.201.26@tcp [ 2379.310755] LNet: Removed LNI 192.168.201.26@tcp2 [ 2379.340614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2379.409571] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2379.418749] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2379.424687] LNet: 116503:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2380.498145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2380.927435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2381.983181] LNet: Removed LNI 192.168.201.26@tcp [ 2382.011440] LNet: Removed LNI 192.168.201.26@tcp2 [ 2382.077208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.186353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.248085] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2383.114518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2383.143307] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2384.207656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2384.241833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2384.267410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2384.319448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2384.760527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2385.823191] LNet: Removed LNI 192.168.201.26@tcp [ 2386.859615] LNet: Removed LNI 192.168.201.26@tcp2 [ 2386.910936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.006863] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2387.015699] LNet: Added LNI 192.168.201.26@tcp2 [8/256/0/180] [ 2387.913465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2396.726619] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2399.221607] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:01:58 (1769756518) [ 2399.363225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2399.383964] LNet: 117617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.388226] LNet: 117617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2399.393909] LNetError: 117617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2399.400181] LNetError: 117617:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2399.417117] LNet: Removed LNI 192.168.201.26@tcp [ 2400.491461] LNet: Removed LNI 192.168.201.26@tcp2 [ 2400.560908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.921538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2400.968503] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2401.829993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2401.853044] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2402.873283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2402.910420] LNet: Removed LNI 192.168.201.26@tcp [ 2402.920269] LNet: Removed LNI 10.1.2.3@tcp [ 2402.941983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.126180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2403.191030] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2403.196892] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2403.858704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2403.879673] LNet: Removed LNI 192.168.201.26@tcp [ 2403.897239] LNet: Removed LNI 10.1.2.3@tcp [ 2412.225606] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:02:11 (1769756531) [ 2412.422556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.457716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.641138] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2412.860291] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2412.870078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2412.877524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2414.280131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2414.355975] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2414.967568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2414.985305] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2417.230995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.26@tcp [ 2417.289072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2417.480153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.26@tcp -m PUT -e local_timeout -r 1 [ 2417.604120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2417.707134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2417.716814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2417.967214] LNetError: 118551:0:(lib-move.c:4947:lnet_parse()) 192.168.201.26@tcp, src 192.168.201.26@tcp: Dropping PUT (error -108 looking up sender) [ 2419.807321] LNet: Removed LNI 192.168.201.26@tcp [ 2419.822741] LNet: Removed LNI 10.1.2.3@tcp [ 2420.745995] Key type .llcrypt unregistered [ 2420.748196] Key type ._llcrypt unregistered [ 2437.045426] Key type ._llcrypt registered [ 2437.051292] Key type .llcrypt registered [ 2438.758158] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:02:38 (1769756558) [ 2439.292541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2439.304929] alg: No test for adler32 (adler32-zlib) [ 2440.228880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2440.530623] LNet: 119859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.065607] Key type .llcrypt unregistered [ 2441.067446] Key type ._llcrypt unregistered [ 2449.688510] Key type ._llcrypt registered [ 2449.697383] Key type .llcrypt registered [ 2450.157050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2450.204461] alg: No test for adler32 (adler32-zlib) [ 2451.192564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2451.233950] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2459.910605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 2460.124504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2461.231449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.126@tcp [ 2461.473448] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 2461.576867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2461.609832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2461.646209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2461.679453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2461.710348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2461.743195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2461.771285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2461.806495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2461.841260] LNet: 120305:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2461.880076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2461.927749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2461.976493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2462.004506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2462.056297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2462.117102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2463.060364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2463.597452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2463.635246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2463.660496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2463.693432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2463.735272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2463.786363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2463.850068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2463.950201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2464.019527] LNet: 120422:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2464.077101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2464.170237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2464.262295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2464.316128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2464.389106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2464.450776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2465.825564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.126@tcp [ 2466.343028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2466.426718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2466.520150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2466.610403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2466.686870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2466.734532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2466.794856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2466.842289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2466.870693] LNet: 120540:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2466.909883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2466.968914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2467.035972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2467.104369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2467.151784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2467.242722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2468.440834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2468.839885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2468.897298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2468.972849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2469.026934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2469.097439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2469.165298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2469.226465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2469.293320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2469.338299] LNet: 120657:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2469.391307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2469.468099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2469.552973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2469.645334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2469.721457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2469.804591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2470.455793] LNet: 120702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2470.459705] LNetError: 120702:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2471.528437] LNet: Removed LNI 192.168.201.26@tcp [ 2471.961918] Key type .llcrypt unregistered [ 2471.963614] Key type ._llcrypt unregistered [ 2492.116973] Key type ._llcrypt registered [ 2492.118701] Key type .llcrypt registered [ 2493.334696] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:03:33 (1769756613) [ 2494.064161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2494.089819] alg: No test for adler32 (adler32-zlib) [ 2495.032801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.642247] LNet: 121587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.534535] Key type .llcrypt unregistered [ 2496.539416] Key type ._llcrypt unregistered [ 2505.460240] Key type ._llcrypt registered [ 2505.468956] Key type .llcrypt registered [ 2505.772641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2505.785448] alg: No test for adler32 (adler32-zlib) [ 2506.800415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2506.853137] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2515.610269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.126@tcp [ 2526.105468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2526.210053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2526.297429] LNet: Added LNI 192.168.201.26@tcp1 [8/256/0/180] [ 2527.410245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2528.270644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.126@tcp [ 2528.415108] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.126@tcp [ 2528.683239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2528.724885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2528.764676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2528.793726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2528.815387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2528.839468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2528.865281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2528.897481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2528.915886] LNet: 122236:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2528.938976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2528.967709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2529.005695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2529.059544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2529.110146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2529.167806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2529.876777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2530.108359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2530.134049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2530.164598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2530.214585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2530.283275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2530.319315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2530.351756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2530.433301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2530.454108] LNet: 122353:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2530.488866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2530.556294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2530.606354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2530.687239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2530.731991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2530.805123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2531.559602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.126@tcp [ 2531.790316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2531.826938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2531.861721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2531.897107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2531.934687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2531.971736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2532.007792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2532.044949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2532.072787] LNet: 122471:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2532.106730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2532.158202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2532.220781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2532.278307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2532.350695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2532.442187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2533.425255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2533.770127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2533.812590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2533.866814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2533.919652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2533.956833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2533.993738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2534.055126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2534.133710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2534.167790] LNet: 122588:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2534.190885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2534.245679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2534.275692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2534.306527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2534.389472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2534.449154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2539.697896] LNet: 122668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2539.708502] LNetError: 122668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.767374] LNet: Removed LNI 192.168.201.26@tcp [ 2540.771679] LNetError: 122668:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2540.795938] LNet: Removed LNI 192.168.201.26@tcp1 [ 2541.360895] Key type .llcrypt unregistered [ 2541.365424] Key type ._llcrypt unregistered [ 2559.743591] Key type ._llcrypt registered [ 2559.746890] Key type .llcrypt registered [ 2561.363841] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:04:41 (1769756681) [ 2570.503547] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:04:50 (1769756690) [ 2579.481850] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:04:59 (1769756699) [ 2589.163907] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:05:08 (1769756708) [ 2600.993887] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:05:20 (1769756720) [ 2611.184083] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:05:31 (1769756731) [ 2620.130245] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:05:39 (1769756739) [ 2631.243555] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:05:51 (1769756751) [ 2641.694480] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:06:01 (1769756761) [ 2652.220623] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:06:12 (1769756772) [ 2662.140960] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:06:21 (1769756781) [ 2673.313976] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:06:33 (1769756793) [ 2683.517137] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:06:43 (1769756803) [ 2692.563813] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:06:52 (1769756812) [ 2701.478185] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:07:01 (1769756821) [ 2709.984845] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:07:10 (1769756830) [ 2719.940120] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:07:19 (1769756839) [ 2729.030296] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:07:28 (1769756848) [ 2729.568557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2729.599493] alg: No test for adler32 (adler32-zlib) [ 2730.627884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2730.821461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2730.847820] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2732.606895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2732.621801] LNet: 133386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2732.630681] LNetError: 133386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2732.642764] LNet: Removed LNI 192.168.201.26@tcp [ 2732.656720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2732.701717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2732.722657] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2742.172769] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:07:41 (1769756861) [ 2752.714911] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:07:52 (1769756872) [ 2762.059540] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:08:02 (1769756882) [ 2772.041982] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:08:11 (1769756891) [ 2781.240514] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:08:21 (1769756901) [ 2790.036966] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:08:29 (1769756909) [ 2800.179315] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:08:39 (1769756919) [ 2811.097960] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:08:50 (1769756930) [ 2821.066602] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:09:01 (1769756941) [ 2830.839888] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:09:10 (1769756950) [ 2840.215871] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:09:20 (1769756960) [ 2850.208502] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:09:29 (1769756969) [ 2859.399286] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2650 sec ========== 02:09:39 (1769756979) [ 2859.864825] LNet: 140859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2859.885131] LNetError: 140859:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2859.899790] LNet: Removed LNI 192.168.201.26@tcp [ 2860.881714] Key type .llcrypt unregistered [ 2860.885366] Key type ._llcrypt unregistered [ 2877.494053] Key type ._llcrypt registered [ 2877.507896] Key type .llcrypt registered [ 2878.144798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2878.164801] alg: No test for adler32 (adler32-zlib) [ 2879.301957] Lustre: Lustre: Build Version: 2.17.50_86_g7a239f5 [ 2879.589782] LNet: Added LNI 192.168.201.26@tcp [8/256/0/180] [ 2881.303301] Key type lgssc registered [ 2883.202687] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2955.357425] Lustre: Mounted lustre-client [ 2961.531559] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2976.421964] LustreError: 143314:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8db34a3ed800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2976.448882] LustreError: 143314:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2976.490204] Lustre: Unmounted lustre-client [ 2999.919695] Key type lgssc unregistered [ 3000.493297] LNet: 143796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3000.503625] LNetError: 143796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3000.532265] LNet: Removed LNI 192.168.201.26@tcp [ 3001.791206] Key type .llcrypt unregistered [ 3001.793984] Key type ._llcrypt unregistered