[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 462920539 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003151] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008487] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010016] pid_max: default: 32768 minimum: 301 [ 0.011163] LSM: Security Framework initializing [ 0.012061] Yama: becoming mindful. [ 0.013042] SELinux: Initializing. [ 0.015071] *** VALIDATE selinux *** [ 0.023342] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027776] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030122] *** VALIDATE tmpfs *** [ 0.032341] *** VALIDATE proc *** [ 0.033264] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.035287] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038027] Spectre V2 : User space: Vulnerable [ 0.039008] Speculative Store Bypass: Vulnerable [ 0.042096] debug: unmapping init [mem 0xffffffff88459000-0xffffffff88460fff] [ 0.044147] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048011] ... generic registers: 4 [ 0.048910] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051016] ... event mask: 000000070000000f [ 0.052326] rcu: Hierarchical SRCU implementation. [ 0.054505] smp: Bringing up secondary CPUs ... [ 0.055613] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.059406] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143025] node 0 deferred pages initialised in 79ms [ 0.147107] devtmpfs: initialized [ 0.148243] x86/mm: Memory block size: 128MB [ 0.152096] gcov: version magic: 0x41383552 [ 0.154399] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161316] pinctrl core: initialized pinctrl subsystem [ 0.164196] [ 0.164902] ************************************************************* [ 0.167016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170016] ** ** [ 0.173022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176014] ** ** [ 0.179014] ** This means that this kernel is built to expose internal ** [ 0.182016] ** IOMMU data structures, which may compromise security on ** [ 0.184014] ** your system. ** [ 0.187018] ** ** [ 0.190014] ** If you see this message and you are not debugging the ** [ 0.193014] ** kernel, report this immediately to your vendor! ** [ 0.194013] ** ** [ 0.197014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199011] ************************************************************* [ 0.202617] NET: Registered protocol family 16 [ 0.204430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210080] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214094] cpuidle: using governor menu [ 0.215725] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219585] PCI: Using configuration type 1 for base access [ 0.222153] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.237057] cryptd: max_cpu_qlen set to 1000 [ 0.241283] ACPI: Added _OSI(Module Device) [ 0.243012] ACPI: Added _OSI(Processor Device) [ 0.244012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.246014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250695] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256373] ACPI: Interpreter enabled [ 0.258061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260015] ACPI: Using IOAPIC for interrupt routing [ 0.262210] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266371] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.276539] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.283021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287094] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.292173] acpiphp: Slot [2] registered [ 0.294070] acpiphp: Slot [5] registered [ 0.296108] acpiphp: Slot [6] registered [ 0.297138] acpiphp: Slot [3] registered [ 0.298146] acpiphp: Slot [4] registered [ 0.299078] acpiphp: Slot [7] registered [ 0.301098] acpiphp: Slot [8] registered [ 0.302093] acpiphp: Slot [9] registered [ 0.304106] acpiphp: Slot [10] registered [ 0.305124] acpiphp: Slot [11] registered [ 0.307100] acpiphp: Slot [12] registered [ 0.308101] acpiphp: Slot [13] registered [ 0.310094] acpiphp: Slot [14] registered [ 0.311104] acpiphp: Slot [15] registered [ 0.313129] acpiphp: Slot [16] registered [ 0.314117] acpiphp: Slot [17] registered [ 0.316097] acpiphp: Slot [18] registered [ 0.317104] acpiphp: Slot [19] registered [ 0.318118] acpiphp: Slot [20] registered [ 0.320112] acpiphp: Slot [21] registered [ 0.322102] acpiphp: Slot [22] registered [ 0.323105] acpiphp: Slot [23] registered [ 0.325131] acpiphp: Slot [24] registered [ 0.326083] acpiphp: Slot [25] registered [ 0.328163] acpiphp: Slot [26] registered [ 0.330106] acpiphp: Slot [27] registered [ 0.332135] acpiphp: Slot [28] registered [ 0.334110] acpiphp: Slot [29] registered [ 0.336127] acpiphp: Slot [30] registered [ 0.337105] acpiphp: Slot [31] registered [ 0.339063] PCI host bridge to bus 0000:00 [ 0.341020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.344026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.347024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.350027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.353026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.356031] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.358166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.359743] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.361930] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.369013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.373054] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.376019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.378014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.379011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381484] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.383625] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388822] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.401017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.406020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.413602] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.450018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.464203] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.484022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.491077] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.508038] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.517129] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.520378] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.523372] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.526371] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.529245] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.534026] iommu: Default domain type: Passthrough [ 0.536407] SCSI subsystem initialized [ 0.538136] ACPI: bus type USB registered [ 0.540129] usbcore: registered new interface driver usbfs [ 0.542081] usbcore: registered new interface driver hub [ 0.544103] usbcore: registered new device driver usb [ 0.547237] pps_core: LinuxPPS API ver. 1 registered [ 0.549014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.552098] PTP clock support registered [ 0.555061] EDAC MC: Ver: 3.0.0 [ 0.556423] PCI: Using ACPI for IRQ routing [ 0.557786] NetLabel: Initializing [ 0.558016] NetLabel: domain hash size = 128 [ 0.559013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.560174] NetLabel: unlabeled traffic allowed by default [ 0.562351] vgaarb: loaded [ 0.564304] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.566020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.573773] clocksource: Switched to clocksource kvm-clock [ 0.700351] VFS: Disk quotas dquot_6.6.0 [ 0.702177] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.705163] *** VALIDATE ramfs *** [ 0.706579] *** VALIDATE hugetlbfs *** [ 0.708957] pnp: PnP ACPI init [ 0.711657] pnp: PnP ACPI: found 6 devices [ 0.734337] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.738400] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.740536] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.742559] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.743896] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.745121] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.747678] NET: Registered protocol family 2 [ 0.749746] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.753993] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.758875] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.765577] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.770703] TCP: Hash tables configured (established 65536 bind 65536) [ 0.773889] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.777402] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.780812] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.784478] NET: Registered protocol family 1 [ 0.787292] RPC: Registered named UNIX socket transport module. [ 0.789859] RPC: Registered udp transport module. [ 0.792479] RPC: Registered tcp transport module. [ 0.794435] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.797099] NET: Registered protocol family 44 [ 0.798824] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.800805] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.803079] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.805787] PCI: CLS 0 bytes, default 64 [ 0.808288] Unpacking initramfs... [ 2.173486] debug: unmapping init [mem 0xffff8dcf3cc64000-0xffff8dcf3ffcffff] [ 2.177696] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.180099] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.183293] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.690449] Initialise system trusted keyrings [ 2.691823] Key type blacklist registered [ 2.693313] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.701775] zbud: loaded [ 2.704351] *** VALIDATE nfs *** [ 2.705260] *** VALIDATE nfs4 *** [ 2.706410] pstore: using deflate compression [ 2.708996] Platform Keyring initialized [ 2.807853] NET: Registered protocol family 38 [ 2.809571] Key type asymmetric registered [ 2.810955] Asymmetric key parser 'x509' registered [ 2.812607] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.815356] io scheduler mq-deadline registered [ 2.817690] io scheduler kyber registered [ 2.819907] io scheduler bfq registered [ 2.823104] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.826830] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.830662] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.835358] ACPI: Power Button [PWRF] [ 2.840750] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.846546] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.855568] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.881973] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.908990] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.913500] Non-volatile memory driver v1.3 [ 2.914677] Linux agpgart interface v0.103 [ 2.945438] virtio_blk virtio1: [vda] 134736 512-byte logical blocks (69.0 MB/65.8 MiB) [ 2.949717] vda: detected capacity change from 0 to 68984832 [ 2.966831] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.969338] vdb: detected capacity change from 0 to 1073741824 [ 2.978323] libphy: Fixed MDIO Bus: probed [ 2.987090] usbcore: registered new interface driver usbserial_generic [ 2.989223] usbserial: USB Serial support registered for generic [ 2.991701] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.995797] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.997645] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.000199] mousedev: PS/2 mouse device common for all mice [ 3.003297] rtc_cmos 00:05: RTC can wake from S4 [ 3.006279] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.010244] rtc_cmos 00:05: registered as rtc0 [ 3.011983] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.013459] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.020041] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.020351] intel_pstate: CPU model not supported [ 3.026956] hid: raw HID events driver (C) Jiri Kosina [ 3.028896] usbcore: registered new interface driver usbhid [ 3.031365] usbhid: USB HID core driver [ 3.033047] drop_monitor: Initializing network drop monitor service [ 3.036234] Initializing XFRM netlink socket [ 3.038865] NET: Registered protocol family 10 [ 3.042623] Segment Routing with IPv6 [ 3.044137] NET: Registered protocol family 17 [ 3.046182] mpls_gso: MPLS GSO support [ 3.053171] RAS: Correctable Errors collector initialized. [ 3.056139] AVX version of gcm_enc/dec engaged. [ 3.057691] AES CTR mode by8 optimization enabled [ 3.143320] sched_clock: Marking stable (3143298626, 0)->(4027850811, -884552185) [ 3.146251] registered taskstats version 1 [ 3.148245] Loading compiled-in X.509 certificates [ 3.149682] zswap: loaded using pool lzo/zbud [ 3.175845] Key type big_key registered [ 3.188609] Key type encrypted registered [ 3.190552] ima: No TPM chip found, activating TPM-bypass! [ 3.193260] ima: Allocated hash algorithm: sha1 [ 3.195403] ima: No architecture policies found [ 3.197235] evm: Initialising EVM extended attributes: [ 3.199763] evm: security.selinux [ 3.201396] evm: security.ima [ 3.202931] evm: security.capability [ 3.204688] evm: HMAC attrs: 0x1 [ 3.207836] rtc_cmos 00:05: setting system clock to 2026-02-18 06:54:39 UTC (1771397679) [ 3.215790] debug: unmapping init [mem 0xffffffff89403000-0xffffffff895fffff] [ 3.219782] debug: unmapping init [mem 0xffffffff88182000-0xffffffff88458fff] [ 3.229067] Write protecting the kernel read-only data: 28672k [ 3.233225] debug: unmapping init [mem 0xffffffff86803000-0xffffffff869fffff] [ 3.236447] debug: unmapping init [mem 0xffffffff87114000-0xffffffff871fffff] [ 3.270231] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.281042] systemd[1]: Detected virtualization kvm. [ 3.283357] systemd[1]: Detected architecture x86-64. [ 3.285815] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.315201] systemd[1]: No hostname configured. [ 3.317441] systemd[1]: Set hostname to . [ 3.320173] random: systemd: uninitialized urandom read (16 bytes read) [ 3.323320] systemd[1]: Initializing machine ID from random generator. [ 3.379392] random: ln: uninitialized urandom read (6 bytes read) [ 3.469627] random: systemd: uninitialized urandom read (16 bytes read) [ 3.471870] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.476384] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.482192] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] [ 3.997375] random: fast init done Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.045198] device-mapper: uevent: version 1.0.3 [ 4.047374] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.633407] virtio_net virtio0 ens2: renamed from eth0 [ 4.692210] scsi host0: ata_piix [ 4.706236] scsi host1: ata_piix [ 4.708162] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.711604] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.283929] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.718535] random: crng init done [ 9.720189] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.025202] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.216417] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.499605] SELinux: Disabled at runtime. [ 11.564203] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.572375] systemd[1]: Detected virtualization kvm. [ 11.574123] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.995475] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.998331] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.003176] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.007308] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.009538] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.015095] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.020470] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ 12.166459] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.403390] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.646735] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.736668] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.758675] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.773603] EDAC sbridge: Ver: 1.1.2 [ 13.743736] Key type dns_resolver registered [ 14.043515] NFS: Registering the id_resolver key type [ 14.045668] Key type id_resolver registered [ 14.047279] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg119-client login: [ 38.201166] hrtimer: interrupt took 6312517 ns [ 85.986446] libcfs: loading out-of-tree module taints kernel. [ 86.080525] Key type ._llcrypt registered [ 86.085168] Key type .llcrypt registered [ 86.781937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 86.803570] alg: No test for adler32 (adler32-zlib) [ 88.257218] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 89.320876] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 91.207170] Key type lgssc registered [ 93.275540] Lustre: Echo OBD driver; http://www.lustre.org/ [ 219.074116] Lustre: Mounted lustre-client [ 223.448592] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.258636] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing check_logdir /tmp/testlogs/ [ 244.705321] Lustre: lustre-OST0000-osc-ffff8dcf88b8b800: disconnect after 23s idle [ 249.952847] Lustre: DEBUG MARKER: oleg119-client.virtnet: executing yml_node [ 254.551028] Lustre: DEBUG MARKER: Client: 2.17.50.152 [ 257.646473] Lustre: DEBUG MARKER: MDS: 2.17.50.152 [ 261.024968] Lustre: DEBUG MARKER: OSS: 2.17.50.152 [ 263.479280] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Feb 18 01:58:57 EST 2026 [ 283.156957] LustreError: 7931:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8dcf88b8b800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 283.195879] Lustre: Unmounted lustre-client [ 313.421338] Key type lgssc unregistered [ 313.729060] LNet: 8380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.736239] LNetError: 8380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.762732] LNet: Removed LNI 192.168.201.19@tcp [ 314.593155] Key type .llcrypt unregistered [ 314.598164] Key type ._llcrypt unregistered [ 344.924359] Key type ._llcrypt registered [ 344.928472] Key type .llcrypt registered [ 345.092285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 345.104671] alg: No test for adler32 (adler32-zlib) [ 346.040226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 346.128257] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 346.160470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 350.459351] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 350.620797] LNet: 9255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.638983] LNetError: 9255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 350.687531] LNet: Removed LNI 192.168.201.19@tcp [ 351.532666] Key type .llcrypt unregistered [ 351.540181] Key type ._llcrypt unregistered [ 358.257276] Key type ._llcrypt registered [ 358.259065] Key type .llcrypt registered [ 359.665870] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 02:00:34 (1771398034) [ 360.302065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 360.332235] alg: No test for adler32 (adler32-zlib) [ 361.273360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.374475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.395438] LNet: 9676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.410973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.447835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 361.460877] LNet: 9683:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 369.714778] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 02:00:44 (1771398044) [ 369.870946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.921837] LNet: 10258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.946955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.050779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 378.509860] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 02:00:53 (1771398053) [ 378.978446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.032909] LNet: 10850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 379.067383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.197670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 385.662398] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 02:01:00 (1771398060) [ 385.796196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.854247] LNet: 11440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.926053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.041436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 392.549529] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 02:01:07 (1771398067) [ 392.885071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.915512] LNet: 12032:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.957879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.106168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 393.741869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 401.986676] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 02:01:16 (1771398076) [ 402.120751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.160351] LNet: 12618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.197302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.310660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 402.808337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 403.542413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 410.545668] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 02:01:25 (1771398085) [ 410.758251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.847439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.919678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 418.332696] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 02:01:33 (1771398093) [ 418.642275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.672668] LNet: 13805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.677667] LNet: 13805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 418.701700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.765731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 418.826028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 419.422522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 419.480350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 419.987779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 420.031235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 420.335559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 420.383603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 420.731690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 420.789812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 421.187692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 421.210850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 421.638180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 421.669551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 422.027222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 422.044748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.371836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 422.412233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 422.746488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 422.771648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 423.260536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 423.284921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 424.026465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 424.082973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 424.721041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 424.778559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 432.672231] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 02:01:47 (1771398107) [ 432.914206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.975393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.048602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 433.080947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 440.785421] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 02:01:55 (1771398115) [ 440.962108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.009099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.045150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 441.065318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 449.251446] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 02:02:03 (1771398123) [ 449.432220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.561856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.768682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 449.813302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 459.740176] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 02:02:14 (1771398134) [ 460.041708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.122986] LNet: 16247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.133311] LNet: 16247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 460.171616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.293299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 460.380772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 468.786190] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 02:02:23 (1771398143) [ 468.997375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.082763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.251711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 469.297603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 476.469678] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 02:02:31 (1771398151) [ 476.841619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.898534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.072446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 477.147800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.886850] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 02:02:40 (1771398160) [ 486.034504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.100210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.316335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 486.911812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.545357] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 02:02:49 (1771398169) [ 494.688688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.787993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.107149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.455616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.113648] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 02:02:57 (1771398177) [ 502.353710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.381800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.501280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.812689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.386398] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 02:03:06 (1771398186) [ 511.479637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.539943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.117977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 512.357568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 519.708681] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 02:03:14 (1771398194) [ 519.788303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.845880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.008437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 520.126754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 527.697815] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 02:03:22 (1771398202) [ 527.923232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.981088] LNet: 21075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.983917] LNet: 21075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 528.060372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.188593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 528.270434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 534.549932] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 02:03:29 (1771398209) [ 534.782686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.828131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.904497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 543.069832] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 02:03:37 (1771398217) [ 543.277907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.306822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.410468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 549.848883] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 02:03:44 (1771398224) [ 550.045317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.110745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.169779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 550.577686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 556.864812] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 02:03:51 (1771398231) [ 557.057434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.130681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.240129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 557.286653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 563.682681] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 02:03:58 (1771398238) [ 563.866417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.924084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.118261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 564.188221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 571.493640] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 02:04:06 (1771398246) [ 571.784149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.832143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.935104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 571.990795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 579.729723] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 02:04:14 (1771398254) [ 579.864534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.925491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.060281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 580.098403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 588.309762] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 02:04:22 (1771398262) [ 588.687439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.736427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.797925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 588.887135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 588.900203] LNetError: 25804:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 588.950854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 588.992887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 595.282441] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 02:04:30 (1771398270) [ 595.369674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.437528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.590817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.s83pliMrTK [ 595.701925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.s83pliMrTK [ 595.762479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.s83pliMrTK [ 601.568817] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 02:04:36 (1771398276) [ 602.011808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.100798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 602.159435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 602.194744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 602.238405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 608.662599] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 02:04:43 (1771398283) [ 608.999509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.045905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.119834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 611.944777] LNet: Message forwarding will be enabled in 2s [ 615.008623] LNet: Message forwarding enabled [ 615.361188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 615.381704] LNet: Message forwarding will stop in 2s to 6s [ 619.859743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 620.930402] LNet: Message forwarding disabled [ 622.675400] LNet: Message forwarding will be enabled in 2s [ 626.079232] LNet: Message forwarding enabled [ 630.508995] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 02:05:05 (1771398305) [ 630.884500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.498357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.619947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 639.425977] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 02:05:14 (1771398314) [ 639.703319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.739639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 639.789788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 639.823080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 639.860374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 639.912595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 639.948663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 639.984816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 640.025973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 640.090868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 640.145718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 640.189676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 640.218784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 640.250532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 640.277467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 640.304504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 640.359524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 640.392307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 640.421635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 640.453701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 640.490378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 640.520614] LNetError: 28874:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 647.149507] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 02:05:21 (1771398321) [ 647.421967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.562879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.616222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.271705] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 02:05:29 (1771398329) [ 656.184089] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg119-server [ 658.120554] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 02:05:33 (1771398333) [ 658.403364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.445087] LNet: 30376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.463025] LNet: 30376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 658.530904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.604180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 658.647927] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 660.295927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[20-24]@tcp [ 660.314116] LNet: 30590:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 660.376105] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 660.421868] LNetError: 30380:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.21@tcp has gone from up to down [ 661.377844] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.20@tcp is being used as a gateway but routing feature is not turned on [ 661.404961] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 662.389891] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 662.404567] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 6 previous similar messages [ 664.422116] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.20@tcp is being used as a gateway but routing feature is not turned on [ 664.450630] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 667.336915] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 02:05:42 (1771398342) [ 668.583538] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 668.599279] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 669.701924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg119-server [ 671.349327] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 02:05:46 (1771398346) [ 671.643742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.680311] LNetError: 30379:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.201.21@tcp [ 671.686690] LNetError: 30389:0:(lib-move.c:4960:lnet_parse()) 192.168.201.20@tcp, src 192.168.201.20@tcp: Dropping REPLY (error -108 looking up sender) [ 671.694748] LNetError: 30389:0:(lib-move.c:4960:lnet_parse()) Skipped 1 previous similar message [ 672.737304] LNetError: 31525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 672.758332] LNet: Removed LNI 192.168.201.19@tcp [ 672.819254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.886431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 672.936345] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 673.776135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[20-25/2]@tcp [ 673.813652] LNet: 31642:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 673.833523] LNet: 31642:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 673.867841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[20-25/2]@tcp [ 673.895596] LNetError: 31529:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.20@tcp has gone from up to down [ 673.920691] LNetError: 31529:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 680.450655] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 02:05:55 (1771398355) [ 680.682872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.706789] LNetError: 32236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.765844] LNet: Removed LNI 192.168.201.19@tcp [ 681.813296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.935911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 682.021888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.060263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.123526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.173596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.200949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.234435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 682.311892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.398590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.476257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 682.556382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.584946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.659050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 682.772776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.848658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.941317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.039801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.095808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.166914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 683.258591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.335272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.404505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.493145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.537636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.618395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 683.711591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.766815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.875925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.947774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.974929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.015882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 684.070753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 684.101887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.165883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 690.877954] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 02:06:05 (1771398365) [ 692.994268] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg119-server [ 694.823971] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 02:06:09 (1771398369) [ 696.501867] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg119-server [ 698.054782] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 02:06:13 (1771398373) [ 698.809622] Key type .llcrypt unregistered [ 698.812331] Key type ._llcrypt unregistered [ 706.075573] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 706.323595] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 706.341936] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 706.353231] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 709.029501] Key type ._llcrypt registered [ 709.034407] Key type .llcrypt registered [ 709.245410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 709.259821] alg: No test for adler32 (adler32-zlib) [ 710.223637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.283750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 710.331280] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 711.128540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 711.153703] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 713.161229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 713.193495] LNet: Removed LNI 10.1.2.3@tcp [ 713.346992] LNet: 34372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.354503] LNetError: 34372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.370494] LNet: Removed LNI 192.168.201.19@tcp [ 713.928735] Key type .llcrypt unregistered [ 713.930538] Key type ._llcrypt unregistered [ 726.200509] Key type ._llcrypt registered [ 726.202474] Key type .llcrypt registered [ 727.380196] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 02:06:42 (1771398402) [ 728.177229] Key type .llcrypt unregistered [ 728.183051] Key type ._llcrypt unregistered [ 734.138407] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.310438] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.332395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.343397] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 736.895573] Key type ._llcrypt registered [ 736.897286] Key type .llcrypt registered [ 737.042390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.056567] alg: No test for adler32 (adler32-zlib) [ 737.984155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.038690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 738.059728] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 738.659597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 738.685883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 740.530545] LNet: 35863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.534384] LNetError: 35863:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.539686] LNet: Removed LNI 192.168.201.19@tcp [ 740.568326] LNet: Removed LNI 10.1.2.3@tcp [ 741.370519] Key type .llcrypt unregistered [ 741.375365] Key type ._llcrypt unregistered [ 752.088630] Key type ._llcrypt registered [ 752.090688] Key type .llcrypt registered [ 753.484832] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 02:07:08 (1771398428) [ 754.105053] Key type .llcrypt unregistered [ 754.109129] Key type ._llcrypt unregistered [ 760.697706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 760.895723] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 760.914210] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 760.920407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 763.599397] Key type ._llcrypt registered [ 763.607518] Key type .llcrypt registered [ 763.883244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 763.910035] alg: No test for adler32 (adler32-zlib) [ 764.900979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.001820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 765.049989] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 767.363590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 767.399773] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 770.014443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 770.063394] LNet: Removed LNI 10.1.2.3@tcp [ 770.082595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 770.137345] LNet: Removed LNI 10.1.2.31@tcp [ 770.164215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 770.221485] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.262728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 771.288572] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 772.465178] LNet: 37786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.473659] LNetError: 37786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 772.485495] LNet: Removed LNI 10.1.2.3@tcp [ 772.503445] LNet: Removed LNI 10.1.2.31@tcp [ 773.291493] Key type .llcrypt unregistered [ 773.295552] Key type ._llcrypt unregistered [ 785.620849] Key type ._llcrypt registered [ 785.622725] Key type .llcrypt registered [ 786.878052] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 02:07:41 (1771398461) [ 787.186968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 787.206628] alg: No test for adler32 (adler32-zlib) [ 788.235889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.321361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 788.371255] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 789.484568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 789.526294] LNet: 38738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.543052] LNetError: 38738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 789.567932] LNet: Removed LNI 192.168.201.19@tcp [ 789.608897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.706104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.19@tcp [ 789.793097] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 790.826960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.19@tcp [ 790.897756] LNet: Removed LNI 192.168.201.19@tcp [ 791.086694] LNet: 38873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.032965] Key type .llcrypt unregistered [ 792.034279] Key type ._llcrypt unregistered [ 807.302621] Key type ._llcrypt registered [ 807.304654] Key type .llcrypt registered [ 808.397630] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 02:08:03 (1771398483) [ 810.502664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg119-server [ 812.287569] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 02:08:07 (1771398487) [ 812.857712] Key type .llcrypt unregistered [ 812.860278] Key type ._llcrypt unregistered [ 819.573409] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 819.825206] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 819.836046] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 819.849253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 822.364678] Key type ._llcrypt registered [ 822.368387] Key type .llcrypt registered [ 822.595553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 822.615897] alg: No test for adler32 (adler32-zlib) [ 823.540150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 823.592851] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 824.424361] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 826.814702] LNet: 40704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.821624] LNetError: 40704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.826584] LNet: Removed LNI 192.168.201.19@tcp1 [ 826.847377] LNet: Removed LNI 10.1.2.3@tcp2 [ 827.508469] Key type .llcrypt unregistered [ 827.511748] Key type ._llcrypt unregistered [ 841.629048] Key type ._llcrypt registered [ 841.634409] Key type .llcrypt registered [ 843.212788] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 02:08:37 (1771398517) [ 843.788473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 843.812239] alg: No test for adler32 (adler32-zlib) [ 844.796249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.854242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 844.877115] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 845.990942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.009250] LNet: 41674:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.026057] LNetError: 41674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.043903] LNet: Removed LNI 192.168.201.19@tcp [ 846.062793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.171635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 846.217269] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 853.578445] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 02:08:48 (1771398528) [ 853.679565] LNet: 42258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.687180] LNetError: 42258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.706356] LNet: Removed LNI 192.168.201.19@tcp [ 854.457446] Key type .llcrypt unregistered [ 854.459928] Key type ._llcrypt unregistered [ 861.297734] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 861.467417] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 861.470435] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 861.473715] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 864.073213] Key type ._llcrypt registered [ 864.074857] Key type .llcrypt registered [ 864.294748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.323078] alg: No test for adler32 (adler32-zlib) [ 865.261292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.359387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.381617] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 866.051605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 866.063977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.462485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.480748] LNet: 43003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.484361] LNetError: 43003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.489322] LNet: Removed LNI 192.168.201.19@tcp [ 868.500048] LNet: Removed LNI 10.1.2.3@tcp [ 868.524720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.612859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 868.632465] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 868.636369] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 869.209423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.246877] LNet: 43030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.252401] LNetError: 43030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.256758] LNetError: 43030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 869.266535] LNet: Removed LNI 192.168.201.19@tcp [ 869.278580] LNet: Removed LNI 10.1.2.3@tcp [ 869.318919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.430110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 869.492933] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 869.504167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.049855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 870.080832] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 871.378209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.424598] LNet: 43192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.431694] LNetError: 43192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.442073] LNetError: 43192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 871.457869] LNet: Removed LNI 192.168.201.19@tcp [ 871.464287] LNet: Removed LNI 10.1.2.3@tcp [ 871.484525] LNet: Removed LNI 192.168.201.19@tcp2 [ 871.540117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.666338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 871.728914] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 871.732805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.737222] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 879.504550] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 02:09:14 (1771398554) [ 879.676426] LNet: 43787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.683119] LNetError: 43787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.692608] LNetError: 43787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 879.701063] LNet: Removed LNI 192.168.201.19@tcp [ 879.708918] LNet: Removed LNI 10.1.2.3@tcp [ 879.732732] LNet: Removed LNI 192.168.201.19@tcp2 [ 880.298862] Key type .llcrypt unregistered [ 880.301266] Key type ._llcrypt unregistered [ 887.239517] Key type ._llcrypt registered [ 887.245548] Key type .llcrypt registered [ 887.537471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 887.556891] alg: No test for adler32 (adler32-zlib) [ 888.589634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.652252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 888.675959] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 889.558743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 889.577981] LNet: 44186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.583920] LNetError: 44186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 889.595746] LNet: Removed LNI 192.168.201.19@tcp [ 889.634443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.688453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 889.730745] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 890.121338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.136987] LNet: 44216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.143919] LNetError: 44216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.156836] LNet: Removed LNI 192.168.201.19@tcp [ 890.175992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.236758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 890.266549] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 890.850601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.874840] LNet: Removed LNI 192.168.201.19@tcp [ 890.928645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.027308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 891.058705] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 891.626512] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 891.735507] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 891.745945] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 891.755955] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 894.263339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.284082] LNet: 44324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.295822] LNet: 44324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 894.301114] LNetError: 44324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.308474] LNetError: 44324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 894.343543] LNet: Removed LNI 192.168.201.19@tcp [ 894.379986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.480611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 894.512091] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 900.521706] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 02:09:35 (1771398575) [ 900.720658] LNet: 44922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.730283] LNetError: 44922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.756150] LNet: Removed LNI 192.168.201.19@tcp [ 901.534947] Key type .llcrypt unregistered [ 901.543666] Key type ._llcrypt unregistered [ 907.583209] Key type ._llcrypt registered [ 907.587748] Key type .llcrypt registered [ 907.803114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 907.826820] alg: No test for adler32 (adler32-zlib) [ 908.729137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.782618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 908.810299] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 909.577141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.592743] LNet: 45321:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.596633] LNetError: 45321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.611810] LNet: Removed LNI 192.168.201.19@tcp [ 909.630723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.712680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 909.757744] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 917.369636] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 02:09:52 (1771398592) [ 917.478319] LNet: 45906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.486727] LNetError: 45906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.505323] LNet: Removed LNI 192.168.201.19@tcp [ 918.205487] Key type .llcrypt unregistered [ 918.209156] Key type ._llcrypt unregistered [ 926.075272] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 926.242144] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 926.256105] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.301770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 928.901073] Key type ._llcrypt registered [ 928.902641] Key type .llcrypt registered [ 929.145239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.164251] alg: No test for adler32 (adler32-zlib) [ 930.141416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.211746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.231846] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 931.194956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.214414] LNet: 46364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.223373] LNetError: 46364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.233669] LNet: Removed LNI 192.168.201.19@tcp [ 931.260658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 931.347592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 931.373687] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 939.026332] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 02:10:13 (1771398613) [ 939.158306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.183725] LNet: 46958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.204057] LNetError: 46958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.216360] LNet: Removed LNI 192.168.201.19@tcp [ 939.253772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.395717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 946.604802] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 02:10:21 (1771398621) [ 946.776905] LNet: 47537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.745285] Key type .llcrypt unregistered [ 947.751968] Key type ._llcrypt unregistered [ 954.341629] Key type ._llcrypt registered [ 954.344529] Key type .llcrypt registered [ 954.486148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.507973] alg: No test for adler32 (adler32-zlib) [ 955.517615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.565595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 955.595955] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 956.382773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.410749] LNet: 47942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.417612] LNetError: 47942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 956.433782] LNet: Removed LNI 192.168.201.19@tcp [ 956.452443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.508730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 956.545603] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 964.704686] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 02:10:38 (1771398638) [ 964.800577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 964.871300] LNet: 48528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 964.883684] LNetError: 48528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 964.913257] LNet: Removed LNI 192.168.201.19@tcp [ 965.000077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.103938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 972.897829] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 02:10:47 (1771398647) [ 974.897365] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 976.945259] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 02:10:51 (1771398651) [ 979.319676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 981.490425] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 02:10:56 (1771398656) [ 983.192808] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 985.440483] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 02:11:00 (1771398660) [ 987.368564] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 989.369441] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 02:11:04 (1771398664) [ 991.455800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 993.459953] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 02:11:08 (1771398668) [ 993.968309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 994.013523] LNet: 50590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.035184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.160856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.210218] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 995.153806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.19 [ 995.179130] LNet: 50723:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 995.186952] LNet: Removed LNI 192.168.201.19@tcp [ 1002.247824] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 02:11:17 (1771398677) [ 1002.567614] LNet: 51293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.577039] LNetError: 51293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.620405] LNet: Removed LNI 192.168.201.19@tcp [ 1003.322977] Key type .llcrypt unregistered [ 1003.325116] Key type ._llcrypt unregistered [ 1010.509897] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1010.805561] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1010.807988] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1010.815532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1013.696584] Key type ._llcrypt registered [ 1013.704580] Key type .llcrypt registered [ 1013.998583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1014.048259] alg: No test for adler32 (adler32-zlib) [ 1015.727388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.220640] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 02:11:38 (1771398698) [ 1024.353471] LNet: 52196:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.372224] LNetError: 52196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1024.392559] LNet: Removed LNI 10.1.2.3@tcp [ 1025.112501] Key type .llcrypt unregistered [ 1025.114935] Key type ._llcrypt unregistered [ 1032.135571] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1032.385189] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1032.392704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1032.399635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1034.106587] Key type ._llcrypt registered [ 1034.111643] Key type .llcrypt registered [ 1034.367847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.391660] alg: No test for adler32 (adler32-zlib) [ 1035.396836] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1041.595546] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 02:11:56 (1771398716) [ 1041.915621] LNet: 53087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.927690] LNetError: 53087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1041.993347] LNet: Removed LNI 10.1.2.3@tcp [ 1042.586651] Key type .llcrypt unregistered [ 1042.590422] Key type ._llcrypt unregistered [ 1049.572216] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1049.865201] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1049.871165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.879628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1052.726908] Key type ._llcrypt registered [ 1052.729870] Key type .llcrypt registered [ 1052.902871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.923538] alg: No test for adler32 (adler32-zlib) [ 1053.841513] LNet: 53392:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1053.851777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.353461] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 02:12:15 (1771398735) [ 1061.473763] LNet: 53978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.488765] LNetError: 53978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1061.516606] LNet: Removed LNI 10.1.2.3@tcp [ 1062.364618] Key type .llcrypt unregistered [ 1062.376144] Key type ._llcrypt unregistered [ 1070.154679] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1070.434527] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1070.436740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1070.442302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1073.241899] Key type ._llcrypt registered [ 1073.247282] Key type .llcrypt registered [ 1073.559428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1073.601325] alg: No test for adler32 (adler32-zlib) [ 1074.691051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.798256] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1087.658404] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 02:12:42 (1771398762) [ 1088.314275] LNet: 55216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.323478] LNetError: 55216:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1088.340886] LNet: Removed LNI 10.1.2.3@tcp [ 1089.278431] Key type .llcrypt unregistered [ 1089.280121] Key type ._llcrypt unregistered [ 1097.420508] Key type ._llcrypt registered [ 1097.423946] Key type .llcrypt registered [ 1097.745989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.782976] alg: No test for adler32 (adler32-zlib) [ 1098.799423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1098.831419] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1107.611094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1108.435988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_interrupt -r 1 [ 1108.482028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_interrupt -r 1 [ 1108.551404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1108.596971] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1108.849402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.869872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1108.898078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.949745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.987982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.033919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.086849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.092848] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1109.153857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.199174] LNet: 55654:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.219392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.269867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.310266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.354060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.416759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.471380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.523088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.916873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_dropped -r 1 [ 1109.949175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_dropped -r 1 [ 1109.978729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1109.999634] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1110.006726] LNet: Skipped 1 previous similar message [ 1110.262506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.335159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.363761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.384164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.410108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.426448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.442019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.456399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.466668] LNet: 55763:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.481073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.508204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.533978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.556341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.576412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.592282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.623822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.093427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_aborted -r 1 [ 1111.123222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_aborted -r 1 [ 1111.151221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1111.169719] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1111.389234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.419545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1111.454342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.479216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.510940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.535957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.559726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.594531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.616630] LNet: 55872:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.631823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.662284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.686161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.724182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.771305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.833268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.883820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.552595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_no_route -r 1 [ 1112.581991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_no_route -r 1 [ 1112.609726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1112.974820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.056297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.142218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.192722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.228294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.266090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.297693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.338576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.471937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.533727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.564818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.602952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.652744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.713306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.755862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.460326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_timeout -r 1 [ 1114.492548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_timeout -r 1 [ 1114.530474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1114.576234] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1114.589376] LNet: Skipped 1 previous similar message [ 1114.818443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.851432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.869574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.894175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.920766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.945461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.999702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.038604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.058460] LNet: 56091:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.063887] LNet: 56091:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 1115.081478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.108702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.134353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.158565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.181793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.223525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.259726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1115.781109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_error -r 1 [ 1115.808927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_error -r 1 [ 1115.835144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1116.040373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.083092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.120174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.145611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.172245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.195888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.217639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.255974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.286183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.322239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.355028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.411304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.468770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.501996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.536415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1116.784958] LNet: 56241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.793206] LNetError: 56241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1117.861161] LNet: Removed LNI 192.168.201.19@tcp [ 1118.353921] Key type .llcrypt unregistered [ 1118.356395] Key type ._llcrypt unregistered [ 1137.967773] Key type ._llcrypt registered [ 1137.971924] Key type .llcrypt registered [ 1139.648401] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 02:13:34 (1771398814) [ 1140.756352] Key type .llcrypt unregistered [ 1140.758085] Key type ._llcrypt unregistered [ 1147.943949] Key type ._llcrypt registered [ 1147.953084] Key type .llcrypt registered [ 1148.272683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.291145] alg: No test for adler32 (adler32-zlib) [ 1149.258469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.285308] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1158.488196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1171.315047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1171.377471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1171.431293] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1173.391890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_interrupt -r 1 [ 1173.444248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_interrupt -r 1 [ 1173.519703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_interrupt -r 1 [ 1173.581327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_interrupt -r 1 [ 1173.628648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_interrupt -r 1 [ 1173.699333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_interrupt -r 1 [ 1173.743347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_interrupt -r 1 [ 1173.814468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_interrupt -r 1 [ 1173.910884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1173.956947] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1173.964669] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1174.251369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.283761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1174.313602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.390694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.468919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.544788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.614728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.736156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.814188] LNet: 57708:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.862912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.946565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.981461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.011804] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1175.019504] LNet: Skipped 2 previous similar messages [ 1175.021203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.069863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.127806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.188895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.868758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_dropped -r 1 [ 1175.918085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_dropped -r 1 [ 1175.974519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_dropped -r 1 [ 1176.024188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_dropped -r 1 [ 1176.064765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_dropped -r 1 [ 1176.091931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_dropped -r 1 [ 1176.135177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_dropped -r 1 [ 1176.183185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_dropped -r 1 [ 1176.222594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1176.244975] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1176.251508] LNet: Skipped 1 previous similar message [ 1176.508747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.544299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.585586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.613681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.645715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.672592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.697974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.725932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.742399] LNet: 57830:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.770512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.806197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.832888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.865405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.900984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.941389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.980223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1177.491625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_aborted -r 1 [ 1177.518537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_aborted -r 1 [ 1177.543275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_aborted -r 1 [ 1177.570332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_aborted -r 1 [ 1177.605348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_aborted -r 1 [ 1177.645971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_aborted -r 1 [ 1177.687882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_aborted -r 1 [ 1177.737899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_aborted -r 1 [ 1177.786890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1178.072955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.132442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1178.182704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.219276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.286464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.335947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.426216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.475156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.502780] LNet: 57951:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.523299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.564391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.598423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.634328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.678713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.717533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.767984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1179.594398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_no_route -r 1 [ 1179.630455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_no_route -r 1 [ 1179.667859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_no_route -r 1 [ 1179.723347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_no_route -r 1 [ 1179.768985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_no_route -r 1 [ 1179.805343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_no_route -r 1 [ 1179.853812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_no_route -r 1 [ 1179.913795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_no_route -r 1 [ 1179.950540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1179.966086] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1179.976982] LNet: Skipped 5 previous similar messages [ 1180.359050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.414960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1180.451877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.486184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.527626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.581493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.642265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.713170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.732156] LNet: 58072:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.750876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.830694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.911794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.966791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.059854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.147980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.186976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.701965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_timeout -r 1 [ 1181.738768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_timeout -r 1 [ 1181.781828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_timeout -r 1 [ 1181.816730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_timeout -r 1 [ 1181.844959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_timeout -r 1 [ 1181.873813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_timeout -r 1 [ 1181.907648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_timeout -r 1 [ 1181.947593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_timeout -r 1 [ 1181.982829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1182.229925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.259085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1182.300129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.346139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.387516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.410937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.431373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.456227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.542352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.556887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.580676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.610412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.635475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.657628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.680331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.121376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_error -r 1 [ 1183.159781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_error -r 1 [ 1183.239850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_error -r 1 [ 1183.281793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_error -r 1 [ 1183.344891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_error -r 1 [ 1183.409118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_error -r 1 [ 1183.459995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_error -r 1 [ 1183.529672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_error -r 1 [ 1183.617064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1184.064268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.192744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1184.299141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.435977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.561921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.654961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.749875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.822911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.879336] LNet: 58315:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.896082] LNet: 58315:0:(api-ni.c:299:recovery_interval_set()) Skipped 1 previous similar message [ 1184.924535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1185.014019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.063804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.123944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.155520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.194650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.228236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.093210] LNet: 58391:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.100523] LNetError: 58391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.107596] LNet: Removed LNI 192.168.201.19@tcp [ 1193.127532] LNet: Removed LNI 192.168.201.19@tcp1 [ 1194.005752] Key type .llcrypt unregistered [ 1194.011407] Key type ._llcrypt unregistered [ 1213.860078] Key type ._llcrypt registered [ 1213.863991] Key type .llcrypt registered [ 1215.649471] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:14:49 (1771398889) [ 1216.538371] Key type .llcrypt unregistered [ 1216.539945] Key type ._llcrypt unregistered [ 1223.498137] Key type ._llcrypt registered [ 1223.499898] Key type .llcrypt registered [ 1223.699552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1223.714803] alg: No test for adler32 (adler32-zlib) [ 1224.618231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1224.653701] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1233.033855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1233.789871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_dropped -r 1 [ 1233.854558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_dropped -r 1 [ 1233.898199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1233.926611] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1234.147716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1234.185935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1234.237610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1234.278287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1234.315464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1234.358371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1234.402646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1234.448240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1234.483440] LNet: 59640:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1234.509369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1234.537507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1234.571513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1234.596081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1234.626640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1234.655972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1234.704667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1235.089767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_error -r 1 [ 1235.114764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_error -r 1 [ 1235.147991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1235.167437] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1235.365416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.411722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1235.434465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.465542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.496805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.531697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.561833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.600668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.619643] LNet: 59750:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.637205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.665141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1235.698141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1235.736924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1235.759989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1235.783737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1235.811510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1236.251504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_timeout -r 1 [ 1236.281437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_timeout -r 1 [ 1236.309325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1236.323150] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1236.499739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.512853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1236.526357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.545985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.560256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.578386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.597259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.615282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.625017] LNet: 59859:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.635092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1236.654711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.672704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.699877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.731220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.756554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.788545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.068508] LNet: 59900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.073541] LNetError: 59900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.094244] LNet: Removed LNI 192.168.201.19@tcp [ 1237.595575] Key type .llcrypt unregistered [ 1237.597233] Key type ._llcrypt unregistered [ 1260.195031] Key type ._llcrypt registered [ 1260.199311] Key type .llcrypt registered [ 1261.807724] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:15:36 (1771398936) [ 1262.846462] Key type .llcrypt unregistered [ 1262.848056] Key type ._llcrypt unregistered [ 1272.129794] Key type ._llcrypt registered [ 1272.133851] Key type .llcrypt registered [ 1272.607494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1272.655125] alg: No test for adler32 (adler32-zlib) [ 1273.835496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1273.931678] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1282.101688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1292.376657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.437351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1292.459709] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1294.005345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_dropped -r 1 [ 1294.081850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e remote_dropped -r 1 [ 1294.128594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_dropped -r 1 [ 1294.199394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e remote_dropped -r 1 [ 1294.257308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e remote_dropped -r 1 [ 1294.336715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e remote_dropped -r 1 [ 1294.419508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e remote_dropped -r 1 [ 1294.493959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e remote_dropped -r 1 [ 1294.550895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1294.574201] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1294.579504] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.119@tcp [ 1294.856718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1294.892437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1294.945980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1294.980373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.038926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.114713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.183981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.224767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.243333] LNet: 61367:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.277168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.335695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.383232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.433498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.458342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.534540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1295.575510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1295.611959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1296.158985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_error -r 1 [ 1296.204615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e remote_error -r 1 [ 1296.240308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_error -r 1 [ 1296.298988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e remote_error -r 1 [ 1296.361679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e remote_error -r 1 [ 1296.432960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e remote_error -r 1 [ 1296.482715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e remote_error -r 1 [ 1296.509541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e remote_error -r 1 [ 1296.537165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1296.557103] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1296.564222] LNet: Skipped 2 previous similar messages [ 1296.817845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1296.847380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1296.871867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1296.898363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1296.920275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1296.944667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1296.969664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.009763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.027984] LNet: 61491:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.044148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.078400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.128643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.181379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.219060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.252587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.287134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1297.312337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.864496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_timeout -r 1 [ 1297.902667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e remote_timeout -r 1 [ 1297.946239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_timeout -r 1 [ 1297.998626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e remote_timeout -r 1 [ 1298.049978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e remote_timeout -r 1 [ 1298.104539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e remote_timeout -r 1 [ 1298.151064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e remote_timeout -r 1 [ 1298.200973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e remote_timeout -r 1 [ 1298.236813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.119@tcp [ 1298.274625] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1298.610717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1298.651820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1298.686935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.724178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.767593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.795984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.825542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.857979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.910200] LNet: 61614:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.938514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.000317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.052249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.108074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.154814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.206302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1299.263406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1299.344386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.669268] LNet: 61692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1304.680546] LNetError: 61692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.684058] LNet: Removed LNI 192.168.201.19@tcp [ 1304.693801] LNet: Removed LNI 192.168.201.19@tcp1 [ 1305.312334] Key type .llcrypt unregistered [ 1305.316879] Key type ._llcrypt unregistered [ 1325.488223] Key type ._llcrypt registered [ 1325.491035] Key type .llcrypt registered [ 1327.204830] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:16:41 (1771399001) [ 1328.076059] Key type .llcrypt unregistered [ 1328.077664] Key type ._llcrypt unregistered [ 1336.486222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1336.626192] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1336.634268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1336.648346] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1339.382599] Key type ._llcrypt registered [ 1339.384789] Key type .llcrypt registered [ 1339.599709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.624811] alg: No test for adler32 (adler32-zlib) [ 1340.571395] LNet: 62770:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1340.594899] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1340.648928] LNet: 62784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1340.660902] LNetError: 62784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.675732] LNet: Removed LNI 192.168.201.19@tcp [ 1341.436524] Key type .llcrypt unregistered [ 1341.437529] Key type ._llcrypt unregistered [ 1348.424191] Key type ._llcrypt registered [ 1348.425762] Key type .llcrypt registered [ 1348.817571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1348.882341] alg: No test for adler32 (adler32-zlib) [ 1349.903763] LNet: 63043:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1349.968963] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1349.971484] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1350.043807] LNet: 63057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.056682] LNetError: 63057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.066462] LNet: Removed LNI 192.168.201.19@tcp [ 1350.079518] LNet: Removed LNI 10.1.2.3@tcp1 [ 1351.294648] Key type .llcrypt unregistered [ 1351.299242] Key type ._llcrypt unregistered [ 1358.807315] Key type ._llcrypt registered [ 1358.814669] Key type .llcrypt registered [ 1359.067737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1359.101821] alg: No test for adler32 (adler32-zlib) [ 1360.095154] LNet: 63317:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1360.135405] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1360.141394] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1360.236973] LNet: 63331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.248368] LNetError: 63331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1360.250952] LNet: Removed LNI 192.168.201.19@tcp [ 1360.276815] LNet: Removed LNI 10.1.2.3@tcp [ 1361.154541] Key type .llcrypt unregistered [ 1361.165775] Key type ._llcrypt unregistered [ 1367.766499] Key type ._llcrypt registered [ 1367.770437] Key type .llcrypt registered [ 1367.963873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1367.981805] alg: No test for adler32 (adler32-zlib) [ 1368.973473] LNet: 63592:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1368.992910] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1368.999426] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1369.085871] LNet: 63606:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.097780] LNetError: 63606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.115434] LNet: Removed LNI 192.168.201.19@tcp [ 1369.142399] LNet: Removed LNI 10.1.2.3@tcp [ 1369.913519] Key type .llcrypt unregistered [ 1369.916720] Key type ._llcrypt unregistered [ 1378.292144] Key type ._llcrypt registered [ 1378.299969] Key type .llcrypt registered [ 1378.644929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1378.674666] alg: No test for adler32 (adler32-zlib) [ 1379.684992] LNet: 63871:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1379.707864] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1379.712906] LNetError: 63871:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1379.730537] LNetError: Error -100 starting up LNI tcp [ 1379.755541] LNet: Removed LNI 192.168.201.19@tcp [ 1380.698950] Key type .llcrypt unregistered [ 1380.700603] Key type ._llcrypt unregistered [ 1395.387875] Key type ._llcrypt registered [ 1395.389393] Key type .llcrypt registered [ 1396.889811] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:17:51 (1771399071) [ 1398.023662] Key type .llcrypt unregistered [ 1398.027954] Key type ._llcrypt unregistered [ 1406.586794] Key type ._llcrypt registered [ 1406.592843] Key type .llcrypt registered [ 1406.799570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1406.816291] alg: No test for adler32 (adler32-zlib) [ 1407.828389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1407.871524] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1416.871599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1418.061515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e network_timeout -r 1 [ 1418.113696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e network_timeout -r 1 [ 1418.185543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1418.225922] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1418.539582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1418.591992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1418.627087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1418.678276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1418.730109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1418.775844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1418.858469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1418.916059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1418.948745] LNet: 65129:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1418.968286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1419.007491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1419.052068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1419.106536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1419.138436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1419.172441] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1419.175587] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1419.181697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1419.609410] LNet: 65168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.617338] LNetError: 65168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.646865] LNet: Removed LNI 192.168.201.19@tcp [ 1421.143674] Key type .llcrypt unregistered [ 1421.146627] Key type ._llcrypt unregistered [ 1441.087360] Key type ._llcrypt registered [ 1441.092942] Key type .llcrypt registered [ 1441.312452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1441.327473] alg: No test for adler32 (adler32-zlib) [ 1442.313333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1442.356667] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1449.643716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1460.133634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.206583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1460.241488] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1461.900619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e network_timeout -r 1 [ 1461.961693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e network_timeout -r 1 [ 1462.025654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e network_timeout -r 1 [ 1462.097092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e network_timeout -r 1 [ 1462.158820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e network_timeout -r 1 [ 1462.227677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e network_timeout -r 1 [ 1462.255343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e network_timeout -r 1 [ 1462.293720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e network_timeout -r 1 [ 1462.345220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1462.387846] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1462.730602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1462.777905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1462.806822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1462.842297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1462.879165] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1462.883125] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.119@tcp [ 1462.884444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1462.930798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1462.957330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1462.992342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1463.018519] LNet: 66047:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1463.045831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1463.077291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1463.110816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.137949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1463.175422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1463.211575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1468.873472] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1469.215794] LNet: 66121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.220814] LNetError: 66121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.239524] LNet: Removed LNI 192.168.201.19@tcp [ 1470.241726] LNetError: 66121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1470.272731] LNet: Removed LNI 192.168.201.19@tcp1 [ 1470.676519] Key type .llcrypt unregistered [ 1470.685882] Key type ._llcrypt unregistered [ 1491.721676] Key type ._llcrypt registered [ 1491.729788] Key type .llcrypt registered [ 1494.153988] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:19:27 (1771399167) [ 1494.658468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1494.690688] alg: No test for adler32 (adler32-zlib) [ 1495.770672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.856786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1495.884228] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1497.162231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1497.195537] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1498.421349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 1498.497714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1498.544549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1498.589757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1498.629480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1498.668840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1498.712795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.19@tcp [ 1498.744335] LNetError: 67223:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.201.19@tcp to recovery [ 1498.911990] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1498.917994] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1500.960367] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1501.138604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1504.031759] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1504.796214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1504.824107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1504.886509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.914198] LNet: 67319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.923747] LNetError: 67319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1504.934984] LNet: Removed LNI 192.168.201.19@tcp [ 1504.955502] LNet: Removed LNI 192.168.201.19@tcp1 [ 1505.024829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.091777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.131040] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1505.821651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1505.837825] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1506.407399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 1506.467628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1506.528839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1506.581989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1506.607703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1506.639082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1506.653989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1506.671933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.19@tcp [ 1506.682710] LNetError: 67592:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.201.19@tcp to recovery [ 1507.103675] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1510.270889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1513.250177] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1513.270983] LNet: Skipped 1 previous similar message [ 1518.513891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1518.556876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1518.590684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1526.989098] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:20:01 (1771399201) [ 1527.119620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.167317] LNet: 68304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.173157] LNetError: 68304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.181499] LNetError: 68304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1527.189956] LNet: Removed LNI 192.168.201.19@tcp [ 1527.205189] LNet: Removed LNI 192.168.201.19@tcp1 [ 1527.270209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1527.354682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1527.378494] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1528.329316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.365967] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1529.848106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 1529.958455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1529.994339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1530.051402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1530.092407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1530.125835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1530.167418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.19@tcp --health 0 [ 1530.317715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1530.404259] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.19@tcp [ 1531.423464] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1531.431339] LNet: Skipped 1 previous similar message [ 1546.260952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1546.504506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1546.566531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1546.643861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.19@tcp --health 500 [ 1546.856444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1553.895899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1553.996261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.014066] LNet: 68802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1554.031691] LNetError: 68802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.034585] LNetError: 68802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1554.046883] LNet: Removed LNI 192.168.201.19@tcp [ 1554.056839] LNet: Removed LNI 192.168.201.19@tcp1 [ 1554.111979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.242673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.289428] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1555.590923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1555.662850] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1557.202838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 1557.327365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1557.474879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1557.576577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1557.636620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1557.725357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1557.785756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1557.837879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.19@tcp --health 0 [ 1558.243401] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1558.258258] LNet: Skipped 3 previous similar messages [ 1559.396276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1568.933923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1569.013278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1569.046482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1575.785972] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:20:50 (1771399250) [ 1576.515673] LNet: 69838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1576.530169] LNetError: 69838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1576.539167] LNetError: 69838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1576.549064] LNet: Removed LNI 192.168.201.19@tcp [ 1576.575766] LNet: Removed LNI 192.168.201.19@tcp1 [ 1577.460960] Key type .llcrypt unregistered [ 1577.462584] Key type ._llcrypt unregistered [ 1584.403954] Key type ._llcrypt registered [ 1584.407656] Key type .llcrypt registered [ 1584.709760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1584.736815] alg: No test for adler32 (adler32-zlib) [ 1585.725922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1585.757353] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1594.126862] Lustre: DEBUG MARKER: Initial discovery [ 1594.394436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.119@tcp [ 1597.049939] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1597.109569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1597.166354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.119@tcp [ 1597.229026] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1597.281903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1597.367309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.119@tcp -r 1 -m GET -l 3 [ 1597.436573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.119@tcp -r 1 -m GET -e local_error [ 1597.502830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.119@tcp -r 1 -m PUT -l 6 [ 1598.049234] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1599.075524] Lustre: DEBUG MARKER: Force oleg119-client.virtnet to discover 192.168.201.119@tcp (in background) [ 1599.307935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.119@tcp [ 1600.725737] Lustre: DEBUG MARKER: Force oleg119-server to discover 192.168.201.19@tcp [ 1602.403657] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1603.280130] Lustre: DEBUG MARKER: Wait for 70600 [ 1604.883491] Lustre: DEBUG MARKER: Finished wait on 70600 [ 1604.952140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1604.996258] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1604.999128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1605.228715] LNet: 71050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.235627] LNetError: 71050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.253508] LNet: Removed LNI 192.168.201.19@tcp [ 1605.680238] Key type .llcrypt unregistered [ 1605.685076] Key type ._llcrypt unregistered [ 1621.814319] Key type ._llcrypt registered [ 1621.825324] Key type .llcrypt registered [ 1623.532923] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:21:37 (1771399297) [ 1624.600434] Key type .llcrypt unregistered [ 1624.611481] Key type ._llcrypt unregistered [ 1633.006750] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1633.249684] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1633.261406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1633.273740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1635.251943] Key type ._llcrypt registered [ 1635.254412] Key type .llcrypt registered [ 1635.576975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1635.601097] alg: No test for adler32 (adler32-zlib) [ 1636.665266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.777691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.813565] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1638.304366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1638.366470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1651.962485] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:22:05 (1771399325) [ 1652.151350] LNet: 73114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.165569] LNetError: 73114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.181335] LNet: Removed LNI 192.168.201.19@tcp [ 1652.228367] LNet: Removed LNI 10.1.2.3@tcp [ 1653.596820] Key type .llcrypt unregistered [ 1653.600957] Key type ._llcrypt unregistered [ 1663.226427] Key type ._llcrypt registered [ 1663.227778] Key type .llcrypt registered [ 1663.510755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1663.546291] alg: No test for adler32 (adler32-zlib) [ 1664.739293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.880107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1664.950100] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1666.095185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1666.150853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1667.957272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.19@tcp [ 1669.030061] LNetError: Unexpected error -22 connecting to 192.168.201.19@tcp at host 192.168.201.19:988 [ 1676.455440] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:22:31 (1771399351) [ 1677.391477] LNet: 74255:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.407700] LNetError: 74255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.419937] LNet: Removed LNI 192.168.201.19@tcp [ 1677.441046] LNet: Removed LNI 10.1.2.3@tcp [ 1678.107959] Key type .llcrypt unregistered [ 1678.109659] Key type ._llcrypt unregistered [ 1685.832778] Key type ._llcrypt registered [ 1685.836075] Key type .llcrypt registered [ 1686.120488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1686.155948] alg: No test for adler32 (adler32-zlib) [ 1687.113526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.185117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1687.223747] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1688.006439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1688.030548] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 1689.555805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.19@tcp1 --nid 192.168.201.19@tcp2 [ 1696.608755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp1 192.168.201.19@tcp2 [ 1696.642464] LNetError: 75091:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.19@tcp1 to 192.168.201.19@tcp2 - Route table may be misconfigured [ 1696.668569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp2 192.168.201.19@tcp1 [ 1707.942375] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:23:01 (1771399381) [ 1708.091091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.141820] LNet: 75658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.154818] LNetError: 75658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.161111] LNet: Removed LNI 192.168.201.19@tcp1 [ 1709.234701] LNet: Removed LNI 192.168.201.19@tcp2 [ 1709.292183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.382307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.431452] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1710.543627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1710.565603] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1711.679665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 1711.758576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.19@tcp [ 1711.810836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp1 192.168.201.19@tcp1 [ 1711.868334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.19@tcp -d 192.168.201.19@tcp -e network_timeout [ 1711.927545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -e network_timeout [ 1711.976210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -e network_timeout [ 1712.052410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -e network_timeout [ 1712.123093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1712.147301] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1712.161575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1712.321395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1712.417270] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1712.480398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1719.077501] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:23:14 (1771399394) [ 1719.671313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1719.690725] LNet: 76526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.696840] LNetError: 76526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.701312] LNetError: 76526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1719.709785] LNet: Removed LNI 192.168.201.19@tcp [ 1720.743209] LNet: Removed LNI 192.168.201.19@tcp1 [ 1720.816887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.922741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1721.371492] LNet: 76563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.046884] Key type .llcrypt unregistered [ 1722.064360] Key type ._llcrypt unregistered [ 1737.145283] Key type ._llcrypt registered [ 1737.156873] Key type .llcrypt registered [ 1739.521583] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:23:33 (1771399413) [ 1740.358560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.386271] alg: No test for adler32 (adler32-zlib) [ 1741.347408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.690053] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1746.367230] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:23:40 (1771399420) [ 1746.526493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.550147] LNet: 77656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.573318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.675812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.700206] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1747.656546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1747.686822] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1749.176716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1749.260435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp1 [ 1749.356159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp1 [ 1759.195220] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:23:52 (1771399432) [ 1761.967955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg119-server [ 1763.540596] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:23:58 (1771399438) [ 1765.692219] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg119-server [ 1767.736959] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:24:02 (1771399442) [ 1770.464942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg119-server [ 1772.488844] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:24:07 (1771399447) [ 1774.646290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg119-server [ 1777.228107] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:24:11 (1771399451) [ 1779.976670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg119-server [ 1781.887809] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:24:16 (1771399456) [ 1784.298356] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg119-server [ 1786.319760] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:24:21 (1771399461) [ 1788.428085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg119-server [ 1791.262215] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:24:25 (1771399465) [ 1794.014045] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg119-server [ 1796.082834] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:24:30 (1771399470) [ 1798.747693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg119-server [ 1801.605858] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:24:35 (1771399475) [ 1801.781564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.856327] LNet: 81534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.865934] LNetError: 81534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.875542] LNet: Removed LNI 192.168.201.19@tcp [ 1802.926676] LNet: Removed LNI 192.168.201.19@tcp1 [ 1802.954693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.045757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.075595] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1804.378834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1804.467463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1804.630768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.689319] LNet: 81690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.697738] LNetError: 81690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1804.711370] LNetError: 81690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1804.728609] LNet: Removed LNI 192.168.201.19@tcp [ 1804.749777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1804.797907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1804.830059] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1805.788788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1805.999448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.019515] LNetError: 81835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.099206] LNet: Removed LNI 192.168.201.19@tcp [ 1807.171116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.306396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1807.357987] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1808.622537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1808.714265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1808.940198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.980612] LNet: 81982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.988797] LNet: 81982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1809.007288] LNetError: 81982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.036543] LNet: Removed LNI 192.168.201.19@tcp [ 1810.108659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.219256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1810.276458] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1810.944646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1812.396163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.478158] LNet: Removed LNI 192.168.201.19@tcp [ 1813.535134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.642668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.676656] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1814.591864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1814.713466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1815.057133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1815.137097] LNet: 82285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1815.154476] LNet: 82285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1815.169576] LNetError: 82285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1815.177288] LNetError: 82285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1815.216339] LNet: Removed LNI 192.168.201.19@tcp [ 1815.247198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.399289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1815.434294] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1816.566731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1816.860859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.893792] LNet: Removed LNI 192.168.201.19@tcp [ 1817.973525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.037861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.074848] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1818.842437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1818.942889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1819.206038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.266659] LNet: Removed LNI 192.168.201.19@tcp [ 1820.290074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.331786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1820.347533] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1821.005151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1821.261433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.309072] LNet: Removed LNI 192.168.201.19@tcp [ 1822.350772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.471665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.513910] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1823.749968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1823.838728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1824.110742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.157337] LNet: 82870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.168606] LNet: 82870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1824.182146] LNetError: 82870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.191809] LNetError: 82870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1824.215324] LNet: Removed LNI 192.168.201.19@tcp [ 1824.240221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.283284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1824.319882] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1825.413706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1825.776430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.857747] LNet: Removed LNI 192.168.201.19@tcp [ 1826.952526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1827.097041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1827.155108] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1828.356444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1828.424234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1829.878383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.956255] LNet: Removed LNI 192.168.201.19@tcp [ 1830.996415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1831.068356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1831.117319] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1832.113310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 1832.362518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.487608] LNet: Removed LNI 192.168.201.19@tcp [ 1833.547775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.647813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.705208] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1834.718890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1834.811700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.878038] LNet: Removed LNI 192.168.201.19@tcp [ 1834.919844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.006472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.038617] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1835.873084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1844.709829] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:25:19 (1771399519) [ 1844.865899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.929388] LNet: 84151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.932370] LNet: 84151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1844.944937] LNetError: 84151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.950516] LNetError: 84151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1844.999451] LNet: Removed LNI 192.168.201.19@tcp [ 1845.062254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1845.192520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1845.241963] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/180] [ 1847.043134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1847.106183] LNet: Removed LNI 192.168.201.19@tcp231 [ 1847.145360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1847.257986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1847.294193] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/180] [ 1847.305456] LNet: 84302:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1848.079726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1848.127960] LNet: Removed LNI 192.168.201.19@tcp231 [ 1848.152247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1848.169232] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/180] [ 1849.908291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.933584] LNet: Removed LNI 192.168.201.19@tcp231 [ 1849.961687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.094497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1850.140656] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/0] [ 1850.146461] LNet: 84454:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1850.781329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1850.838301] LNet: Removed LNI 192.168.201.19@tcp231 [ 1850.888407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1850.935772] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/0] [ 1852.358895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.388331] LNet: Removed LNI 192.168.201.19@tcp231 [ 1852.402241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.456385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1852.478325] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/60] [ 1852.489287] LNet: 84606:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1853.047352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1853.107061] LNet: Removed LNI 192.168.201.19@tcp231 [ 1853.137175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1853.168781] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/60] [ 1854.217824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.233814] LNet: Removed LNI 192.168.201.19@tcp231 [ 1854.241227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.279052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1854.299524] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/180] [ 1854.650282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1854.691941] LNet: Removed LNI 192.168.201.19@tcp231 [ 1854.707110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1854.732869] LNet: Added LNI 192.168.201.19@tcp231 [8/256/0/180] [ 1864.421208] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:25:39 (1771399539) [ 1864.948682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.036802] LNet: Removed LNI 192.168.201.19@tcp231 [ 1865.101538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.244872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1865.322080] LNet: Added LNI 192.168.201.19@tcp232 [8/256/0/180] [ 1867.196886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1867.250784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1867.292990] LNet: Removed LNI 192.168.201.19@tcp232 [ 1867.307530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1867.327764] LNet: Added LNI 192.168.201.19@tcp232 [8/256/0/180] [ 1868.508284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1868.575777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1868.612158] LNet: Removed LNI 192.168.201.19@tcp232 [ 1868.636328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1868.692391] LNet: Added LNI 192.168.201.19@tcp232 [8/256/0/180] [ 1869.717635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1869.753556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1869.792282] LNet: Removed LNI 192.168.201.19@tcp232 [ 1869.810529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1869.852075] LNet: Added LNI 192.168.201.19@tcp232 [8/256/0/180] [ 1870.692883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1870.730741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1870.782444] LNet: Removed LNI 192.168.201.19@tcp232 [ 1870.798477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1879.535292] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:25:53 (1771399553) [ 1880.245885] LNet: 86598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1880.257278] LNet: 86598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1881.294193] Key type .llcrypt unregistered [ 1881.299853] Key type ._llcrypt unregistered [ 1890.278854] Key type ._llcrypt registered [ 1890.279882] Key type .llcrypt registered [ 1890.479929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1890.507616] alg: No test for adler32 (adler32-zlib) [ 1891.461617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1891.481272] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1900.491238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 1913.528213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.612636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1913.659560] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1915.892977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_interrupt -r 2 [ 1915.912571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_interrupt -r 2 [ 1915.946735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_interrupt -r 2 [ 1915.971825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_interrupt -r 2 [ 1916.006879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_interrupt -r 2 [ 1916.040262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_interrupt -r 2 [ 1916.071524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_interrupt -r 2 [ 1916.103212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_interrupt -r 2 [ 1916.131337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.119@tcp [ 1916.148154] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1916.153728] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1916.385899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1916.426415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1916.459238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1916.483864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1916.507297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1916.537803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1916.564626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1916.708469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1916.751741] LNet: 87263:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1916.793259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1916.846249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1916.940082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1916.970434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1917.002206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1917.082298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1918.225968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_dropped -r 2 [ 1918.307051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_dropped -r 2 [ 1918.389667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_dropped -r 2 [ 1918.527384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_dropped -r 2 [ 1918.656766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_dropped -r 2 [ 1918.762912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_dropped -r 2 [ 1918.910384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_dropped -r 2 [ 1919.007874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_dropped -r 2 [ 1919.086688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.119@tcp [ 1919.146743] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1919.726170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1919.818144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1919.916639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1919.990803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1920.081830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1920.179916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1920.279594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1920.345597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1920.413100] LNet: 87384:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.466875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1920.559512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1920.698359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1920.840984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1920.941846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1921.071452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1922.293621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_aborted -r 2 [ 1922.352771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_aborted -r 2 [ 1922.446692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_aborted -r 2 [ 1922.521671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_aborted -r 2 [ 1922.620097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_aborted -r 2 [ 1922.752748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_aborted -r 2 [ 1922.826880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_aborted -r 2 [ 1922.896880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_aborted -r 2 [ 1922.938563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.119@tcp [ 1922.973763] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1923.322343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1923.359684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1923.394869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1923.447440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1923.530294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1923.599051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1923.630368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1923.667024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1923.682709] LNet: 87505:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1923.701493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1923.736589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1923.766861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1923.807258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1923.857305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1923.942111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1924.807216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_no_route -r 2 [ 1924.909413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_no_route -r 2 [ 1924.985668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_no_route -r 2 [ 1925.052359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_no_route -r 2 [ 1925.157578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_no_route -r 2 [ 1925.207135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_no_route -r 2 [ 1925.278805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_no_route -r 2 [ 1925.333785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_no_route -r 2 [ 1925.382695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.119@tcp [ 1925.403754] LNet: There was an unexpected network error while writing to 192.168.201.119: rc = -22 [ 1925.620434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1925.652212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1925.678917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1925.714113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1925.750583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1925.790613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1925.856704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1925.920602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1925.945458] LNet: 87625:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1925.982961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1926.005683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1926.033931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1926.063462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1926.106249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1926.147728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1926.620784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e local_timeout -r 2 [ 1926.645022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e local_timeout -r 2 [ 1926.671492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e local_timeout -r 2 [ 1926.701955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e local_timeout -r 2 [ 1926.733756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e local_timeout -r 2 [ 1926.759504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e local_timeout -r 2 [ 1926.778220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e local_timeout -r 2 [ 1926.801922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e local_timeout -r 2 [ 1926.828439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.119@tcp [ 1927.182318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1927.223587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1927.255712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1927.306774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1927.342673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1927.402267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1927.453910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1927.492489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1927.529849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1927.601868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1927.657770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1927.719515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1927.796528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1927.849247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1928.522942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp -m GET -e remote_dropped -r 2 [ 1928.565528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp -d 192.168.201.119@tcp1 -m GET -e remote_dropped -r 2 [ 1928.611438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp -m GET -e remote_dropped -r 2 [ 1928.645843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp -d 192.168.201.19@tcp1 -m GET -e remote_dropped -r 2 [ 1928.693175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp -m GET -e remote_dropped -r 2 [ 1928.755523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.119@tcp1 -m GET -e remote_dropped -r 2 [ 1928.808776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp -m GET -e remote_dropped -r 2 [ 1928.884698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.19@tcp1 -d 192.168.201.19@tcp1 -m GET -e remote_dropped -r 2 [ 1928.932556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.19@tcp 192.168.201.119@tcp [ 1928.964956] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.119@tcp [ 1929.391116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1929.424548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1929.495503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1929.542638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1929.602457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1929.655628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1929.713948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1929.770460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1929.819970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1929.893750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1929.945564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1929.994652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1930.043268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1930.085534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1937.133558] LNet: 87941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.139400] LNetError: 87941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.143888] LNet: Removed LNI 192.168.201.19@tcp [ 1937.151155] LNet: Removed LNI 192.168.201.19@tcp1 [ 1937.557955] Key type .llcrypt unregistered [ 1937.559601] Key type ._llcrypt unregistered [ 1955.288808] Key type ._llcrypt registered [ 1955.291415] Key type .llcrypt registered [ 1956.675736] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:27:11 (1771399631) [ 1957.244264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1957.260378] alg: No test for adler32 (adler32-zlib) [ 1958.219230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.284972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1958.313414] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1958.950353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1958.973733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.19@tcp [ 1958.990357] LNetError: 88921:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.201.19@tcp to recovery [ 1959.068902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.19@tcp [ 1959.163734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1965.187591] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:27:20 (1771399640) [ 1965.381811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1965.436686] LNet: 89511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.447853] LNetError: 89511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.473818] LNet: Removed LNI 192.168.201.19@tcp [ 1965.493949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.539345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1965.585630] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1966.724165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1966.775664] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 1967.984469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.19@tcp [ 1968.020511] LNetError: 89777:0:(api-ni.c:4353:lnet_ni_set_healthv()) manually adding local NI 192.168.201.19@tcp to recovery [ 1968.608804] LNet: 1 local NIs in recovery (showing 1): 192.168.201.19@tcp [ 1968.627795] LNet: There was an unexpected network error while writing to 192.168.201.19: rc = -22 [ 1969.307669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.19@tcp [ 1978.348996] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:27:33 (1771399653) [ 1980.358836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg119-server [ 1981.926599] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:27:37 (1771399657) [ 1983.633559] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg119-server [ 1985.737500] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:27:40 (1771399660) [ 1988.065406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg119-server [ 1990.373608] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:27:44 (1771399664) [ 1990.502026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1990.553807] LNet: 91403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.562573] LNetError: 91403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1990.567347] LNet: Removed LNI 192.168.201.19@tcp [ 1990.576185] LNet: Removed LNI 192.168.201.19@tcp1 [ 1990.602843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1990.680621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.703265] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 1991.799774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1991.883238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1991.954516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1992.065692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1992.142709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1992.225443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1999.541939] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:27:54 (1771399674) [ 1999.620461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1999.668441] LNet: 92127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.683675] LNetError: 92127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1999.697437] LNetError: 92127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1999.718170] LNet: Removed LNI 192.168.201.19@tcp [ 1999.755346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.809860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1999.830492] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2007.377919] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:28:02 (1771399682) [ 2007.488763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2007.539565] LNet: 92832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.552939] LNetError: 92832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.578276] LNet: Removed LNI 192.168.201.19@tcp [ 2007.619429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.677454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.694871] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2015.367442] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:28:10 (1771399690) [ 2016.789514] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2018.409958] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:28:13 (1771399693) [ 2018.905392] LNet: 93873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.910286] LNetError: 93873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.923448] LNet: Removed LNI 192.168.201.19@tcp [ 2019.391396] Key type .llcrypt unregistered [ 2019.393335] Key type ._llcrypt unregistered [ 2025.782762] Key type ._llcrypt registered [ 2025.786858] Key type .llcrypt registered [ 2025.988783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.012516] alg: No test for adler32 (adler32-zlib) [ 2026.965591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2027.007392] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2035.395557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 2040.498324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.119@tcp [ 2041.111194] LNet: 94290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.120801] LNetError: 94290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.143154] LNet: Removed LNI 192.168.201.19@tcp [ 2041.736746] Key type .llcrypt unregistered [ 2041.744580] Key type ._llcrypt unregistered [ 2056.709267] Key type ._llcrypt registered [ 2056.711986] Key type .llcrypt registered [ 2059.022975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2061.268705] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2063.616594] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:28:57 (1771399737) [ 2063.982724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2064.002611] alg: No test for adler32 (adler32-zlib) [ 2065.223984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2065.447641] LNet: 95378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.922403] Key type .llcrypt unregistered [ 2065.924458] Key type ._llcrypt unregistered [ 2072.917466] Key type ._llcrypt registered [ 2072.921332] Key type .llcrypt registered [ 2073.141503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2073.167981] alg: No test for adler32 (adler32-zlib) [ 2074.070525] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2074.073602] LNet: 95634:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2074.084354] LNet: 95634:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2074.123629] LNetError: 95646:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.21@tcp is being used as a gateway but routing feature is not turned on [ 2074.141192] LNetError: 95646:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.21@tcp has gone from up to down [ 2074.693619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2074.739114] LNet: 95692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2074.750264] LNetError: 95692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2074.771308] LNet: Removed LNI 192.168.201.19@tcp [ 2074.819878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.915559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2075.000487] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2075.011717] LNet: 95699:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2075.022914] LNet: 95699:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 2075.031227] LNetError: 95699:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.21@tcp is a gateway. Can not delete it [ 2075.038571] LNetError: 95699:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.201.21@tcp is a router. Can not be deleted [ 2075.063328] LNet: 95699:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2075.088579] LNetError: 95696:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.23@tcp is being used as a gateway but routing feature is not turned on [ 2075.122054] LNetError: 95696:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 2075.136577] LNetError: 95696:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.201.23@tcp has gone from up to down [ 2075.148198] LNetError: 95696:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2081.904711] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:29:16 (1771399756) [ 2083.813290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg119-server [ 2085.799610] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:29:20 (1771399760) [ 2087.847611] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg119-server [ 2089.837491] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:29:24 (1771399764) [ 2091.484867] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 2093.535235] Key type lgssc registered [ 2094.710717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2115.512857] Key type lgssc unregistered [ 2115.870633] LNet: 98120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.875790] LNetError: 98120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2115.898427] LNet: Removed LNI 192.168.201.19@tcp [ 2116.546080] Key type .llcrypt unregistered [ 2116.557744] Key type ._llcrypt unregistered [ 2132.747688] Key type ._llcrypt registered [ 2132.750218] Key type .llcrypt registered [ 2134.174395] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:30:08 (1771399808) [ 2134.842156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2134.873720] alg: No test for adler32 (adler32-zlib) [ 2135.848259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.905249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2135.947942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2136.007774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2136.087047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2143.489525] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:30:18 (1771399818) [ 2143.686961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.745620] LNet: 99521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.798437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.935552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.965049] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2145.146966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.165521] LNet: 99671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2145.170825] LNetError: 99671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.208984] LNet: Removed LNI 192.168.201.19@tcp [ 2145.254384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.316367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2145.348723] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2145.876442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.923961] LNetError: 99699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.940812] LNet: Removed LNI 192.168.201.19@tcp [ 2145.969653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.045602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2146.077947] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2154.863829] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:30:29 (1771399829) [ 2155.188654] LNet: 100291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.191961] LNet: 100291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2155.198275] LNetError: 100291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.211980] LNet: Removed LNI 192.168.201.19@tcp [ 2155.959470] Key type .llcrypt unregistered [ 2155.963364] Key type ._llcrypt unregistered [ 2162.810485] Key type ._llcrypt registered [ 2162.817109] Key type .llcrypt registered [ 2163.163695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2163.211977] alg: No test for adler32 (adler32-zlib) [ 2164.182879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.248252] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2164.250415] LNet: *** cfs_fail_loc=e003, val=100*** [ 2164.885546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 2164.910101] LNet: *** cfs_fail_loc=e003, val=100*** [ 2164.912580] LNet: Skipped 2 previous similar messages [ 2164.922268] Ping reply received with mapping: [ 2164.925106] found metadata reply NI[192.168.201.19@tcp] [ 2164.929575] 13 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2164.932274] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2164.935152] 5a 5a 5a 5a 5a 5a 5a 5a [ 2165.045925] LNet: 100688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.050729] LNetError: 100688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2166.132711] LNet: Removed LNI 192.168.201.19@tcp [ 2167.008712] Key type .llcrypt unregistered [ 2167.011641] Key type ._llcrypt unregistered [ 2180.044328] Key type ._llcrypt registered [ 2180.047901] Key type .llcrypt registered [ 2181.758936] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:30:56 (1771399856) [ 2182.536723] Key type .llcrypt unregistered [ 2182.539755] Key type ._llcrypt unregistered [ 2192.316821] Key type ._llcrypt registered [ 2192.320633] Key type .llcrypt registered [ 2192.722873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2192.779798] alg: No test for adler32 (adler32-zlib) [ 2194.300733] LNetError: 101735:0:(api-ni.c:2734:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2194.925589] Key type .llcrypt unregistered [ 2194.927135] Key type ._llcrypt unregistered [ 2201.758622] Key type ._llcrypt registered [ 2201.760784] Key type .llcrypt registered [ 2203.463465] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:31:17 (1771399877) [ 2204.069653] Key type .llcrypt unregistered [ 2204.072274] Key type ._llcrypt unregistered [ 2210.914187] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2210.917705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2213.489216] Key type ._llcrypt registered [ 2213.494166] Key type .llcrypt registered [ 2213.729233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2213.741978] alg: No test for adler32 (adler32-zlib) [ 2214.701815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.793045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2226.307314] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:31:40 (1771399900) [ 2226.449138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2226.499553] LNet: 103508:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2226.505533] LNetError: 103508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2226.533570] LNet: Removed LNI 10.1.2.3@tcp [ 2226.603486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2227.473699] Key type .llcrypt unregistered [ 2227.476465] Key type ._llcrypt unregistered [ 2234.875958] Key type ._llcrypt registered [ 2234.880191] Key type .llcrypt registered [ 2235.143411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2235.156993] alg: No test for adler32 (adler32-zlib) [ 2236.115916] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2236.187173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 2236.386996] LNet: 103804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.426684] LNetError: 103804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.451171] LNet: Removed LNI 192.168.201.19@tcp [ 2237.280619] Key type .llcrypt unregistered [ 2237.284984] Key type ._llcrypt unregistered [ 2249.835611] Key type ._llcrypt registered [ 2249.838521] Key type .llcrypt registered [ 2251.322925] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:32:05 (1771399925) [ 2252.072952] Key type .llcrypt unregistered [ 2252.083173] Key type ._llcrypt unregistered [ 2259.200392] Key type ._llcrypt registered [ 2259.201990] Key type .llcrypt registered [ 2259.593795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2259.636119] alg: No test for adler32 (adler32-zlib) [ 2269.218684] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:32:23 (1771399943) [ 2269.298421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.387734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.490454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2269.544497] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2270.532653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2270.569868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2270.598569] LNet: Removed LNI 192.168.201.19@tcp [ 2271.108769] LNet: 105549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.754627] Key type .llcrypt unregistered [ 2271.760621] Key type ._llcrypt unregistered [ 2284.527730] Key type ._llcrypt registered [ 2284.530904] Key type .llcrypt registered [ 2285.918162] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:32:40 (1771399960) [ 2286.644602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.682592] alg: No test for adler32 (adler32-zlib) [ 2287.591822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.637863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2287.657106] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2288.616930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 2288.731996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.19@tcp [ 2288.779076] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2288.789773] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2288.792022] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2288, 0, 0 [ 2288.805228] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2288.810651] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2288.815340] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2288.825175] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2288, 0, 0 [ 2288.833534] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2288.836409] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2288.849782] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2288.852145] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2288, 0, 2338 [ 2288.864553] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2288.870278] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2288.874629] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2288.879515] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2288, 0, 2338 [ 2288.887494] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2296.811854] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:32:51 (1771399971) [ 2297.621898] LNet: 107090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.630317] LNetError: 107090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.651782] LNet: Removed LNI 192.168.201.19@tcp [ 2298.451281] Key type .llcrypt unregistered [ 2298.454376] Key type ._llcrypt unregistered [ 2307.262434] Key type ._llcrypt registered [ 2307.264954] Key type .llcrypt registered [ 2307.550130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2307.596842] alg: No test for adler32 (adler32-zlib) [ 2308.652053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2308.688086] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2318.200516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 2329.273531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2329.328449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2329.364055] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 2330.663881] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2330.894311] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2330.905899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2330.908398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2333.286644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2333.318331] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2349.335426] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:33:42 (1771400022) [ 2349.536073] LNet: 108596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.552468] LNetError: 108596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.624348] LNet: Removed LNI 192.168.201.19@tcp [ 2350.629760] LNetError: 108596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2350.644513] LNet: Removed LNI 192.168.201.19@tcp1 [ 2350.694376] LNet: Removed LNI 10.1.2.3@tcp99 [ 2351.445381] Key type .llcrypt unregistered [ 2351.447709] Key type ._llcrypt unregistered [ 2360.120420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2360.128537] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2363.079480] Key type ._llcrypt registered [ 2363.082137] Key type .llcrypt registered [ 2363.419559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2363.451911] alg: No test for adler32 (adler32-zlib) [ 2364.422541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2364.491283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2364.531820] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2365.050218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2365.067883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2366.754631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2366.848426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2366.928992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2366.978938] LNetError: 109344:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2367.034946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2367.096948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2367.960242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2368.119474] LNet: 109364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.140410] LNetError: 109364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.183183] LNet: Removed LNI 192.168.201.19@tcp [ 2369.189437] LNetError: 109364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2369.217606] LNet: Removed LNI 10.1.2.3@tcp [ 2370.442621] Key type .llcrypt unregistered [ 2370.447248] Key type ._llcrypt unregistered [ 2386.872343] Key type ._llcrypt registered [ 2386.875443] Key type .llcrypt registered [ 2388.707197] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:34:22 (1771400062) [ 2389.184773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2389.194609] alg: No test for adler32 (adler32-zlib) [ 2390.253181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2390.315502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2390.338291] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2391.269769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.19@tcp [ 2391.397520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg119-client@tcp [ 2399.630278] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:34:34 (1771400074) [ 2399.848933] LNet: 110898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2399.862753] LNetError: 110898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.945770] LNet: Removed LNI 192.168.201.19@tcp [ 2401.793969] Key type .llcrypt unregistered [ 2401.796886] Key type ._llcrypt unregistered [ 2409.166211] Key type ._llcrypt registered [ 2409.169821] Key type .llcrypt registered [ 2409.355614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2409.368772] alg: No test for adler32 (adler32-zlib) [ 2410.425735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2410.480990] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2410.562297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2418.565944] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:34:53 (1771400093) [ 2420.442598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2422.484645] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:34:57 (1771400097) [ 2424.236553] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2426.194660] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:35:00 (1771400100) [ 2426.591230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.638388] LNet: 112318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.651051] LNetError: 112318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.680371] LNet: Removed LNI 192.168.201.19@tcp [ 2426.712455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.827699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2426.909742] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2427.905633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2427.997846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2437.289408] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:35:11 (1771400111) [ 2437.425698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2437.462688] LNet: 113018:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.475881] LNetError: 113018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.534169] LNet: Removed LNI 192.168.201.19@tcp [ 2437.581583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2437.699995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2437.733726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2438.533262] Key type .llcrypt unregistered [ 2438.541106] Key type ._llcrypt unregistered [ 2451.828832] Key type ._llcrypt registered [ 2451.831825] Key type .llcrypt registered [ 2453.334154] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:35:27 (1771400127) [ 2454.013827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2454.023786] alg: No test for adler32 (adler32-zlib) [ 2455.049190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2455.144676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2455.190887] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2455.987773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2456.042387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2463.563558] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:35:38 (1771400138) [ 2463.783650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2463.809935] LNet: 114584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.817918] LNetError: 114584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2463.848830] LNet: Removed LNI 192.168.201.19@tcp [ 2463.882818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2463.931065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2463.975817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2470.850528] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:35:45 (1771400145) [ 2471.209937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2471.243283] LNet: 115166:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2471.271671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2471.369528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2471.910631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2471.943684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2472.000873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2472.385602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2472.426425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2472.533127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2472.551365] LNet: 115199:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.572939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.654783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2473.305515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2473.376867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2473.462343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2473.938783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2473.998276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2481.551384] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:35:56 (1771400156) [ 2481.674259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.716392] LNet: 115793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2481.757172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.929212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2481.992503] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2482.988404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2483.008819] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2484.108743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2484.189224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2484.577338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2484.616963] LNetError: 116079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2484.625446] LNet: Removed LNI 192.168.201.19@tcp [ 2484.636499] LNet: Removed LNI 192.168.201.19@tcp2 [ 2484.694677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.823836] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2484.838208] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2484.842775] LNet: 116085:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2485.547772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2485.771577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2485.789806] LNet: 116122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.793905] LNet: 116122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2485.798918] LNetError: 116122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2485.803151] LNetError: 116122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2486.815557] LNet: Removed LNI 192.168.201.19@tcp [ 2486.826781] LNet: Removed LNI 192.168.201.19@tcp2 [ 2486.861280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2486.908801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2486.926241] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2487.631106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2487.651691] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2488.823619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2488.858749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2488.907672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2489.166783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2489.212069] LNetError: 116407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2489.219807] LNetError: 116407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2489.226263] LNet: Removed LNI 192.168.201.19@tcp [ 2490.278988] LNet: Removed LNI 192.168.201.19@tcp2 [ 2490.308498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2490.380852] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2490.387200] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2490.391568] LNet: 116413:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2491.120610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2491.440641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.479758] LNet: Removed LNI 192.168.201.19@tcp [ 2492.523217] LNet: Removed LNI 192.168.201.19@tcp2 [ 2492.580845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.667357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2492.705099] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2493.631641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2493.645716] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2494.624880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2494.705416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2495.066470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2495.103916] LNet: 116730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2495.111323] LNet: 116730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2495.117533] LNetError: 116730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2495.128236] LNetError: 116730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2495.135452] LNet: Removed LNI 192.168.201.19@tcp [ 2495.154258] LNet: Removed LNI 192.168.201.19@tcp2 [ 2495.231239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.335723] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2495.344406] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2495.357442] LNet: 116736:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2496.027535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2496.288797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2497.375192] LNet: Removed LNI 192.168.201.19@tcp [ 2497.395659] LNet: Removed LNI 192.168.201.19@tcp2 [ 2497.435767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.506423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2497.532759] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2498.345697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2498.379347] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2499.153277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2499.181921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2499.229539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2499.468603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2500.511417] LNet: Removed LNI 192.168.201.19@tcp [ 2501.544758] LNet: Removed LNI 192.168.201.19@tcp2 [ 2501.575686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.647756] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2501.654320] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2501.659116] LNet: 117064:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2502.383559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2502.605252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2502.622101] LNet: Removed LNI 192.168.201.19@tcp [ 2503.661922] LNet: Removed LNI 192.168.201.19@tcp2 [ 2503.734841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2503.823231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2503.853948] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2504.540157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2504.559424] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2505.633023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2505.674351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2505.756383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2506.188670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2506.214339] LNetError: 117398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2506.219925] LNetError: 117398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2506.230320] LNet: Removed LNI 192.168.201.19@tcp [ 2506.238405] LNet: Removed LNI 192.168.201.19@tcp2 [ 2506.252610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2506.334703] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2506.339301] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2506.343456] LNet: 117404:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2507.136154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2507.550152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2508.575399] LNet: Removed LNI 192.168.201.19@tcp [ 2508.600166] LNet: Removed LNI 192.168.201.19@tcp2 [ 2508.679988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2508.748751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2508.812519] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2509.728840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2509.749756] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2510.740913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2510.786194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2510.813537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2510.885202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2511.293540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2511.314857] LNet: 117758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2511.319338] LNet: 117758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2512.351709] LNet: Removed LNI 192.168.201.19@tcp [ 2513.401179] LNet: Removed LNI 192.168.201.19@tcp2 [ 2513.463216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.555322] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2513.565501] LNet: Added LNI 192.168.201.19@tcp2 [8/256/0/180] [ 2514.147693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2521.621411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2523.269567] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:36:38 (1771400198) [ 2523.656773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2523.678564] LNetError: 118525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2523.683530] LNetError: 118525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2523.687775] LNet: Removed LNI 192.168.201.19@tcp [ 2524.716448] LNet: Removed LNI 192.168.201.19@tcp2 [ 2524.802820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2525.174437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2525.202389] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2525.999405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2526.020235] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2526.976484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2526.994432] LNet: Removed LNI 192.168.201.19@tcp [ 2527.011737] LNet: Removed LNI 10.1.2.3@tcp [ 2527.035777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2527.154567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2527.205878] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2527.225252] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2527.834625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2527.864845] LNet: Removed LNI 192.168.201.19@tcp [ 2527.883245] LNet: Removed LNI 10.1.2.3@tcp [ 2535.264843] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:36:49 (1771400209) [ 2535.403592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2535.487341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2535.672602] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2535.878277] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2535.893285] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2535.902306] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2538.269323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2538.322716] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2538.943590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2538.966388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2541.901596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.19@tcp [ 2541.948947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2542.032414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.19@tcp -m PUT -e local_timeout -r 1 [ 2542.155942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2542.246723] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2542.256966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2542.620226] LNetError: 119464:0:(lib-move.c:4960:lnet_parse()) 192.168.201.19@tcp, src 192.168.201.19@tcp: Dropping PUT (error -108 looking up sender) [ 2543.649358] LNet: 119923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2543.652300] LNet: 119923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2544.735192] LNet: Removed LNI 192.168.201.19@tcp [ 2544.756464] LNet: Removed LNI 10.1.2.3@tcp [ 2545.196985] Key type .llcrypt unregistered [ 2545.202951] Key type ._llcrypt unregistered [ 2558.364867] Key type ._llcrypt registered [ 2558.366312] Key type .llcrypt registered [ 2560.084100] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:37:14 (1771400234) [ 2560.555816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2560.585322] alg: No test for adler32 (adler32-zlib) [ 2561.553256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2562.027715] LNet: 120776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2562.630412] Key type .llcrypt unregistered [ 2562.634396] Key type ._llcrypt unregistered [ 2569.119201] Key type ._llcrypt registered [ 2569.126485] Key type .llcrypt registered [ 2569.335828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2569.347831] alg: No test for adler32 (adler32-zlib) [ 2570.328761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2570.352552] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2577.933642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 2578.256913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2579.019631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.119@tcp [ 2579.310458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2579.408502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2579.450495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2579.489245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2579.551132] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.119@tcp [ 2579.610765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2579.693756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2579.740334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2579.797665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2579.825470] LNet: 121226:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2579.873742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2579.935160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2580.025229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2580.085789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2580.172223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2580.229860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2581.351433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2581.761796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2581.807905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2581.859309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2581.909141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2581.984645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2582.051386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2582.125614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2582.163048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2582.183229] LNet: 121343:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2582.204833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2582.230883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2582.261912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2582.288367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2582.318181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2582.354961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2583.129913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.119@tcp [ 2583.383988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2583.423141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2583.462937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2583.495878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2583.526397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2583.564685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2583.596513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2583.634689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2583.651483] LNet: 121461:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2583.670514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2583.714972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2583.760519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2583.803302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2583.848957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2583.882517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2584.522842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2584.849776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2584.892754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2584.941556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2585.000201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2585.027224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2585.063455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2585.109595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2585.147512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2585.181090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2585.219961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2585.242319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2585.277681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2585.311550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2585.354235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2585.709790] LNet: 121624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2585.716719] LNetError: 121624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2585.730457] LNet: Removed LNI 192.168.201.19@tcp [ 2586.189262] Key type .llcrypt unregistered [ 2586.191335] Key type ._llcrypt unregistered [ 2603.114090] Key type ._llcrypt registered [ 2603.117947] Key type .llcrypt registered [ 2604.499606] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:37:59 (1771400279) [ 2604.877511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2604.896957] alg: No test for adler32 (adler32-zlib) [ 2605.862542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2606.353883] LNet: 122513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.089470] Key type .llcrypt unregistered [ 2607.093368] Key type ._llcrypt unregistered [ 2613.895805] Key type ._llcrypt registered [ 2613.899840] Key type .llcrypt registered [ 2614.248283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2614.308852] alg: No test for adler32 (adler32-zlib) [ 2615.378503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2615.445930] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2623.383748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.119@tcp [ 2632.292764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2632.357053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2632.386431] LNet: Added LNI 192.168.201.19@tcp1 [8/256/0/180] [ 2633.712663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2634.364771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.119@tcp [ 2634.609454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2634.650467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2634.696896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2634.736509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2634.784374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2634.826715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2634.859614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2634.894094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2634.917817] LNet: 123166:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2634.918829] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.119@tcp [ 2634.939292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2634.969698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2635.004622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2635.032831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2635.061674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2635.087800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2635.831386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2636.127345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2636.157212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2636.183904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2636.207384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2636.230453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2636.270707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2636.316386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2636.356113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2636.392786] LNet: 123284:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2636.420419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2636.479803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2636.521336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2636.570735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2636.605471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2636.647932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2637.465722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.119@tcp [ 2637.903446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2637.937890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2638.001693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2638.092224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2638.186551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2638.253604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2638.292561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2638.314257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2638.327845] LNet: 123401:0:(api-ni.c:299:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2638.350598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2638.415431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2638.473227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2638.542939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2638.592470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2638.643592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2639.468474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2639.780840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2639.815088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2639.844556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2639.877253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2639.913800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2639.949291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2639.989040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2640.025960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2640.070165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2640.154033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2640.199332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2640.252519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2640.300430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2640.371056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2645.408882] LNet: 123599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.423699] LNetError: 123599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.434412] LNet: Removed LNI 192.168.201.19@tcp [ 2645.446501] LNet: Removed LNI 192.168.201.19@tcp1 [ 2646.113580] Key type .llcrypt unregistered [ 2646.115161] Key type ._llcrypt unregistered [ 2666.343751] Key type ._llcrypt registered [ 2666.354826] Key type .llcrypt registered [ 2668.192391] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:39:02 (1771400342) [ 2676.508387] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:39:11 (1771400351) [ 2683.817826] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:39:18 (1771400358) [ 2691.326891] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:39:26 (1771400366) [ 2699.934249] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:39:34 (1771400374) [ 2709.499674] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:39:43 (1771400383) [ 2718.432154] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:39:53 (1771400393) [ 2726.734284] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:40:01 (1771400401) [ 2734.226201] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:40:08 (1771400408) [ 2740.763399] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:40:15 (1771400415) [ 2747.969655] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:40:22 (1771400422) [ 2755.479572] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:40:30 (1771400430) [ 2763.169582] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:40:37 (1771400437) [ 2771.287103] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:40:45 (1771400445) [ 2781.407609] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:40:55 (1771400455) [ 2789.933734] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:41:04 (1771400464) [ 2799.345173] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:41:13 (1771400473) [ 2807.761802] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:41:22 (1771400482) [ 2808.103651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2808.125375] alg: No test for adler32 (adler32-zlib) [ 2809.043535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2809.152364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2809.175955] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2810.372567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2810.395561] LNet: 134314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2810.404342] LNetError: 134314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2810.417074] LNet: Removed LNI 192.168.201.19@tcp [ 2810.440541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2810.522830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2810.547828] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2817.999784] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:41:32 (1771400492) [ 2825.120868] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:41:40 (1771400500) [ 2833.265986] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:41:48 (1771400508) [ 2841.863142] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:41:56 (1771400516) [ 2850.090406] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:42:04 (1771400524) [ 2859.161963] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:42:13 (1771400533) [ 2867.166294] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:42:22 (1771400542) [ 2875.426789] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:42:30 (1771400550) [ 2883.064763] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:42:38 (1771400558) [ 2890.858310] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:42:45 (1771400565) [ 2898.070820] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:42:52 (1771400572) [ 2906.180638] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:43:00 (1771400580) [ 2913.250748] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2649 sec ========== 02:43:08 (1771400588) [ 2913.578301] LNet: 141792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2913.583583] LNetError: 141792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2913.597770] LNet: Removed LNI 192.168.201.19@tcp [ 2914.309716] Key type .llcrypt unregistered [ 2914.313599] Key type ._llcrypt unregistered [ 2925.045078] Key type ._llcrypt registered [ 2925.050585] Key type .llcrypt registered [ 2925.576915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2925.586685] alg: No test for adler32 (adler32-zlib) [ 2926.747855] Lustre: Lustre: Build Version: 2.17.50_152_g9bf97e3 [ 2927.143692] LNet: Added LNI 192.168.201.19@tcp [8/256/0/180] [ 2928.983259] Key type lgssc registered [ 2931.019326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3005.619190] Lustre: Mounted lustre-client [ 3011.256650] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3024.064936] LustreError: 144253:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8dcf89f59000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3024.116809] Lustre: Unmounted lustre-client [ 3062.041637] Key type lgssc unregistered [ 3062.368369] LNet: 144738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3062.378849] LNetError: 144738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3062.395719] LNet: Removed LNI 192.168.201.19@tcp [ 3063.134493] Key type .llcrypt unregistered [ 3063.136799] Key type ._llcrypt unregistered