[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 486065228 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003096] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008352] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.012050] LSM: Security Framework initializing [ 0.013042] Yama: becoming mindful. [ 0.015004] SELinux: Initializing. [ 0.016084] *** VALIDATE selinux *** [ 0.024673] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029008] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032105] *** VALIDATE tmpfs *** [ 0.033411] *** VALIDATE proc *** [ 0.034226] *** VALIDATE cgroup *** [ 0.035007] *** VALIDATE cgroup2 *** [ 0.037057] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038147] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041006] Speculative Store Bypass: Vulnerable [ 0.044086] debug: unmapping init [mem 0xffffffff8f259000-0xffffffff8f260fff] [ 0.046783] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047717] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049009] ... bit width: 48 [ 0.049953] ... generic registers: 4 [ 0.050010] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053008] ... event mask: 000000070000000f [ 0.054291] rcu: Hierarchical SRCU implementation. [ 0.056532] smp: Bringing up secondary CPUs ... [ 0.057603] x86: Booting SMP configuration: [ 0.058018] .... node #0, CPUs: #1 #2 #3 [ 0.061156] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138869] node 0 deferred pages initialised in 72ms [ 0.142163] devtmpfs: initialized [ 0.143277] x86/mm: Memory block size: 128MB [ 0.145958] gcov: version magic: 0x41383552 [ 0.149367] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152129] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.155335] pinctrl core: initialized pinctrl subsystem [ 0.156204] [ 0.156704] ************************************************************* [ 0.159011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161011] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166011] ** ** [ 0.168010] ** This means that this kernel is built to expose internal ** [ 0.170011] ** IOMMU data structures, which may compromise security on ** [ 0.172011] ** your system. ** [ 0.174011] ** ** [ 0.177009] ** If you see this message and you are not debugging the ** [ 0.179012] ** kernel, report this immediately to your vendor! ** [ 0.181010] ** ** [ 0.183011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.185009] ************************************************************* [ 0.187724] NET: Registered protocol family 16 [ 0.189506] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.192057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.198119] cpuidle: using governor menu [ 0.199702] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202397] PCI: Using configuration type 1 for base access [ 0.205156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.214059] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216030] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.220029] cryptd: max_cpu_qlen set to 1000 [ 0.222258] ACPI: Added _OSI(Module Device) [ 0.224018] ACPI: Added _OSI(Processor Device) [ 0.225014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.227036] ACPI: Added _OSI(Processor Aggregator Device) [ 0.232234] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237436] ACPI: Interpreter enabled [ 0.239051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240015] ACPI: Using IOAPIC for interrupt routing [ 0.241121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243342] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.253421] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.255069] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.258029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.261133] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.265710] acpiphp: Slot [2] registered [ 0.267164] acpiphp: Slot [5] registered [ 0.269108] acpiphp: Slot [6] registered [ 0.270102] acpiphp: Slot [3] registered [ 0.272105] acpiphp: Slot [4] registered [ 0.273122] acpiphp: Slot [7] registered [ 0.274086] acpiphp: Slot [8] registered [ 0.276080] acpiphp: Slot [9] registered [ 0.277086] acpiphp: Slot [10] registered [ 0.278087] acpiphp: Slot [11] registered [ 0.280091] acpiphp: Slot [12] registered [ 0.281103] acpiphp: Slot [13] registered [ 0.283095] acpiphp: Slot [14] registered [ 0.284093] acpiphp: Slot [15] registered [ 0.286116] acpiphp: Slot [16] registered [ 0.287123] acpiphp: Slot [17] registered [ 0.289095] acpiphp: Slot [18] registered [ 0.290113] acpiphp: Slot [19] registered [ 0.291058] acpiphp: Slot [20] registered [ 0.292116] acpiphp: Slot [21] registered [ 0.294082] acpiphp: Slot [22] registered [ 0.295094] acpiphp: Slot [23] registered [ 0.297134] acpiphp: Slot [24] registered [ 0.298162] acpiphp: Slot [25] registered [ 0.300124] acpiphp: Slot [26] registered [ 0.302157] acpiphp: Slot [27] registered [ 0.303079] acpiphp: Slot [28] registered [ 0.304090] acpiphp: Slot [29] registered [ 0.306083] acpiphp: Slot [30] registered [ 0.307084] acpiphp: Slot [31] registered [ 0.309077] PCI host bridge to bus 0000:00 [ 0.310021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.312024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.319025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.321036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.323181] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326662] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.331293] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338686] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.341328] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.344019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348039] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.351768] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.354094] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.356049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.358943] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.363996] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.375023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.380021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.385692] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.403023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414052] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.422653] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.455028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.465911] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.468383] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.470445] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.473542] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.475219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.481039] iommu: Default domain type: Passthrough [ 0.482879] SCSI subsystem initialized [ 0.483286] ACPI: bus type USB registered [ 0.486196] usbcore: registered new interface driver usbfs [ 0.489431] usbcore: registered new interface driver hub [ 0.493320] usbcore: registered new device driver usb [ 0.496348] pps_core: LinuxPPS API ver. 1 registered [ 0.499015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.504172] PTP clock support registered [ 0.507135] EDAC MC: Ver: 3.0.0 [ 0.509110] PCI: Using ACPI for IRQ routing [ 0.512476] NetLabel: Initializing [ 0.514013] NetLabel: domain hash size = 128 [ 0.516014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.517108] NetLabel: unlabeled traffic allowed by default [ 0.520046] vgaarb: loaded [ 0.521375] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.523017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.532368] clocksource: Switched to clocksource kvm-clock [ 0.656056] VFS: Disk quotas dquot_6.6.0 [ 0.657454] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.659780] *** VALIDATE ramfs *** [ 0.661647] *** VALIDATE hugetlbfs *** [ 0.663823] pnp: PnP ACPI init [ 0.667888] pnp: PnP ACPI: found 6 devices [ 0.693657] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.697528] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.699809] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.701856] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.704523] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.706838] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.709458] NET: Registered protocol family 2 [ 0.711663] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.716489] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.719519] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.740688] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.793254] TCP: Hash tables configured (established 65536 bind 65536) [ 0.797578] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.800827] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803451] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806029] NET: Registered protocol family 1 [ 0.808719] RPC: Registered named UNIX socket transport module. [ 0.810795] RPC: Registered udp transport module. [ 0.812385] RPC: Registered tcp transport module. [ 0.813892] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.815641] NET: Registered protocol family 44 [ 0.816909] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.818884] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.821187] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.823607] PCI: CLS 0 bytes, default 64 [ 0.825552] Unpacking initramfs... [ 2.355309] debug: unmapping init [mem 0xffff9655bcc64000-0xffff9655bffcffff] [ 2.361424] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.363653] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.367658] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.880833] Initialise system trusted keyrings [ 2.882463] Key type blacklist registered [ 2.884393] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.893701] zbud: loaded [ 2.896745] *** VALIDATE nfs *** [ 2.897890] *** VALIDATE nfs4 *** [ 2.899448] pstore: using deflate compression [ 2.903249] Platform Keyring initialized [ 2.996759] NET: Registered protocol family 38 [ 2.997976] Key type asymmetric registered [ 2.999454] Asymmetric key parser 'x509' registered [ 3.000888] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.006143] io scheduler mq-deadline registered [ 3.007780] io scheduler kyber registered [ 3.009271] io scheduler bfq registered [ 3.011086] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.013386] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.015472] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.017420] ACPI: Power Button [PWRF] [ 3.021452] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.027104] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.036386] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.064076] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.091173] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.095880] Non-volatile memory driver v1.3 [ 3.097403] Linux agpgart interface v0.103 [ 3.127698] virtio_blk virtio1: [vda] 134632 512-byte logical blocks (68.9 MB/65.7 MiB) [ 3.130268] vda: detected capacity change from 0 to 68931584 [ 3.143847] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.146486] vdb: detected capacity change from 0 to 1073741824 [ 3.152344] libphy: Fixed MDIO Bus: probed [ 3.163335] usbcore: registered new interface driver usbserial_generic [ 3.164782] usbserial: USB Serial support registered for generic [ 3.166574] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.169933] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.171465] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.173740] mousedev: PS/2 mouse device common for all mice [ 3.175887] rtc_cmos 00:05: RTC can wake from S4 [ 3.178465] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.179802] rtc_cmos 00:05: registered as rtc0 [ 3.183501] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.186630] intel_pstate: CPU model not supported [ 3.188890] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.192888] hid: raw HID events driver (C) Jiri Kosina [ 3.193521] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.195406] usbcore: registered new interface driver usbhid [ 3.200965] usbhid: USB HID core driver [ 3.202633] drop_monitor: Initializing network drop monitor service [ 3.204808] Initializing XFRM netlink socket [ 3.206991] NET: Registered protocol family 10 [ 3.211562] Segment Routing with IPv6 [ 3.213378] NET: Registered protocol family 17 [ 3.215409] mpls_gso: MPLS GSO support [ 3.221966] RAS: Correctable Errors collector initialized. [ 3.224355] AVX version of gcm_enc/dec engaged. [ 3.225876] AES CTR mode by8 optimization enabled [ 3.317280] sched_clock: Marking stable (3317125586, 0)->(4174913098, -857787512) [ 3.320729] registered taskstats version 1 [ 3.323256] Loading compiled-in X.509 certificates [ 3.325748] zswap: loaded using pool lzo/zbud [ 3.352141] Key type big_key registered [ 3.366060] Key type encrypted registered [ 3.368055] ima: No TPM chip found, activating TPM-bypass! [ 3.370255] ima: Allocated hash algorithm: sha1 [ 3.371884] ima: No architecture policies found [ 3.373709] evm: Initialising EVM extended attributes: [ 3.375754] evm: security.selinux [ 3.376993] evm: security.ima [ 3.378045] evm: security.capability [ 3.379423] evm: HMAC attrs: 0x1 [ 3.381982] rtc_cmos 00:05: setting system clock to 2026-05-04 15:51:57 UTC (1777909917) [ 3.388546] debug: unmapping init [mem 0xffffffff90203000-0xffffffff903fffff] [ 3.391798] debug: unmapping init [mem 0xffffffff8ef82000-0xffffffff8f258fff] [ 3.404112] Write protecting the kernel read-only data: 28672k [ 3.406131] debug: unmapping init [mem 0xffffffff8d603000-0xffffffff8d7fffff] [ 3.407873] debug: unmapping init [mem 0xffffffff8df14000-0xffffffff8dffffff] [ 3.433851] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.439641] systemd[1]: Detected virtualization kvm. [ 3.441672] systemd[1]: Detected architecture x86-64. [ 3.443246] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.469644] systemd[1]: No hostname configured. [ 3.471059] systemd[1]: Set hostname to . [ 3.473401] random: systemd: uninitialized urandom read (16 bytes read) [ 3.476245] systemd[1]: Initializing machine ID from random generator. [ 3.521542] random: ln: uninitialized urandom read (6 bytes read) [ 3.593407] random: systemd: uninitialized urandom read (16 bytes read) [ 3.595634] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.600639] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.604452] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.206460] device-mapper: uevent: version 1.0.3 [ 4.208531] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ 5.023384] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ 5.046099] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.135846] scsi host0: ata_piix [ 5.262473] scsi host1: ata_piix [ 5.263957] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.266360] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.863207] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.891512] random: crng init done [ 9.893152] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.970169] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.132761] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.387140] SELinux: Disabled at runtime. [ 12.442765] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.452528] systemd[1]: Detected virtualization kvm. [ 12.454654] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.907378] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.910863] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.916569] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.920539] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.924069] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.932280] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.938877] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ 13.025443] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting POSIX Message Queue File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.437995] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.743201] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.764438] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.974049] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.988530] EDAC sbridge: Ver: 1.1.2 [ 15.231416] Key type dns_resolver registered [ 15.529889] NFS: Registering the id_resolver key type [ 15.531782] Key type id_resolver registered [ 15.533195] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg122-client login: [ 64.068285] libcfs: loading out-of-tree module taints kernel. [ 64.115965] Key type ._llcrypt registered [ 64.117651] Key type .llcrypt registered [ 64.591242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 64.599148] alg: No test for adler32 (adler32-zlib) [ 65.714533] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 66.224243] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 67.920201] Key type lgssc registered [ 68.708445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 133.350754] Lustre: Mounted lustre-client [ 136.148774] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.524532] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing check_logdir /tmp/testlogs/ [ 143.594154] Lustre: DEBUG MARKER: oleg122-client.virtnet: executing yml_node [ 145.751305] Lustre: DEBUG MARKER: Client: 2.17.52.127 [ 147.105425] Lustre: DEBUG MARKER: MDS: 2.17.52.127 [ 148.497348] Lustre: DEBUG MARKER: OSS: 2.17.52.127 [ 149.286709] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon May 4 11:54:22 EDT 2026 [ 156.108797] LustreError: 7904:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff965606355000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 156.130092] Lustre: Unmounted lustre-client [ 165.755465] Key type lgssc unregistered [ 165.917841] LNet: 8349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 165.922281] LNetError: 8349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 165.934135] LNet: Removed LNI 192.168.201.22@tcp [ 166.304388] Key type .llcrypt unregistered [ 166.306604] Key type ._llcrypt unregistered [ 178.414295] Key type ._llcrypt registered [ 178.416245] Key type .llcrypt registered [ 178.581671] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 178.593172] alg: No test for adler32 (adler32-zlib) [ 179.426263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 179.479673] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 179.490811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 181.479163] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 181.546224] LNet: 9225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 181.552982] LNetError: 9225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 181.565505] LNet: Removed LNI 192.168.201.22@tcp [ 181.827665] Key type .llcrypt unregistered [ 181.829434] Key type ._llcrypt unregistered [ 184.513981] Key type ._llcrypt registered [ 184.517971] Key type .llcrypt registered [ 185.226135] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:54:58 (1777910098) [ 185.423953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 185.432956] alg: No test for adler32 (adler32-zlib) [ 186.260893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.299165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.310175] LNet: 9637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 186.321846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.355693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 186.370280] LNet: 9644:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 189.561160] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:55:02 (1777910102) [ 189.612276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.636236] LNet: 10216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 189.661533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.719145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 193.002849] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:55:06 (1777910106) [ 193.061065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.082350] LNet: 10797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.098162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.172265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 196.362602] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:55:09 (1777910109) [ 196.484454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.517322] LNet: 11389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 196.530444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.591967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 199.761567] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:55:13 (1777910113) [ 199.837796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.861508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.907904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 200.083465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 203.249293] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:55:16 (1777910116) [ 203.378105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 203.390248] LNet: 12562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 203.393469] LNet: 12562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 203.408707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 203.441754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 203.617800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 203.823931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 207.174589] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:55:20 (1777910120) [ 207.255698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 207.304486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 207.351784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 211.297256] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:55:24 (1777910124) [ 211.339248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.367527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.431203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 211.457168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 211.703630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 211.727788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 211.993940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 212.018939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 212.223066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 212.247404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 212.444338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 212.470241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 212.641728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 212.665368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 212.869561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 212.893701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 213.067114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 213.091361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 213.284823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 213.311701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 213.508225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 213.528848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 213.717543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 213.737823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 213.906950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 213.926689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 214.088593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 214.111155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 217.031708] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:55:30 (1777910130) [ 217.114815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 217.141988] LNet: 14421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 217.144681] LNet: 14421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 217.155574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 217.207727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 217.230520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 220.660928] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:55:34 (1777910134) [ 220.713339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.739390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.781726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 220.800363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 223.732697] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:55:37 (1777910137) [ 223.788106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.807501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.847184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 223.866627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 226.817256] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 11:55:40 (1777910140) [ 226.875472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 226.899226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 226.945362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 226.964536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 230.047581] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 11:55:43 (1777910143) [ 230.113532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.145381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 230.182414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 230.200773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 233.145760] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 11:55:46 (1777910146) [ 233.218183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.231647] LNet: 17347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.234542] LNet: 17347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 233.246520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 233.284919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 233.307075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 236.330493] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 11:55:49 (1777910149) [ 236.372334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.405386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.474942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 236.709704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 239.820987] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 11:55:53 (1777910153) [ 239.853643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 239.873378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 239.955388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 240.049827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 242.886095] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 11:55:56 (1777910156) [ 242.915180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 242.932364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 242.984906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 243.065101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 245.916950] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 11:55:59 (1777910159) [ 245.957681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 245.979129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 246.160220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 246.275079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 249.058462] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 11:56:02 (1777910162) [ 249.116434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 249.140407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 249.186828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 249.212517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 252.178148] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 11:56:05 (1777910165) [ 252.233766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.257464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.310890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 252.339631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 255.236599] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 11:56:08 (1777910168) [ 255.280988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.299190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.331717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 258.025726] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 11:56:11 (1777910171) [ 258.072634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.099855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.151974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 261.234763] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 11:56:14 (1777910174) [ 261.273288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 261.292369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 261.334712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 261.525883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 264.685429] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 11:56:18 (1777910178) [ 264.725192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 264.748542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 264.796824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 264.815871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 267.872845] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 11:56:21 (1777910181) [ 267.909988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 267.921607] LNet: 23950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 267.924926] LNet: 23950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 267.936459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 267.974514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 267.990910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 270.691161] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 11:56:24 (1777910184) [ 270.724878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.745179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.784399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 270.803473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 273.682245] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 11:56:27 (1777910187) [ 273.735840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.761696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.815319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 273.840387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 276.568214] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 11:56:30 (1777910190) [ 276.600864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.618934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.651704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 276.681161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 276.691432] LNetError: 25726:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 276.708911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 276.726907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 279.225918] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 11:56:32 (1777910192) [ 279.276972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.296283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.335697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.nCNPOLWNZP [ 279.364989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.nCNPOLWNZP [ 279.390813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.nCNPOLWNZP [ 281.644542] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 11:56:35 (1777910195) [ 281.738477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 281.759599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.789373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 281.804705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 281.825343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 284.644023] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 11:56:38 (1777910198) [ 284.682544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 284.699726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 284.731545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 285.830636] LNet: Message forwarding will be enabled in 2s [ 287.907048] LNet: Message forwarding enabled [ 287.984418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 287.994910] LNet: Message forwarding will stop in 2s to 6s [ 292.189698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 292.698515] LNet: Message forwarding disabled [ 293.231150] LNet: Message forwarding will be enabled in 2s [ 296.402933] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 11:56:49 (1777910209) [ 296.471813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.139689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.184175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 299.987791] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 11:56:53 (1777910213) [ 300.040712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 300.057712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.087132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 300.106768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 300.122673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 300.138890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 300.160417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 300.174668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 300.191859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 300.210941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 300.229703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 300.248177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 300.264980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 300.279560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 300.295900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 300.310971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 300.324760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 300.344602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 300.364758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 300.393648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 300.411738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 300.424487] LNetError: 28783:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 303.119765] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 11:56:56 (1777910216) [ 303.204381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.237785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.290989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 306.661123] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 11:57:00 (1777910220) [ 307.458618] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg122-server [ 308.324649] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 11:57:01 (1777910221) [ 308.382345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.405812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.435955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 308.446992] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 309.177911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[23-27]@tcp [ 309.189275] LNet: 30497:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 309.217209] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 309.229726] LNetError: 30287:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.26@tcp has gone from up to down [ 310.248261] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.24@tcp is being used as a gateway but routing feature is not turned on [ 310.254032] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 311.270837] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 311.282721] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 312.293372] LNetError: 30292:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.23@tcp has gone from up to down [ 312.297678] LNetError: 30292:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 312.370178] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 11:57:05 (1777910225) [ 313.208165] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg122-server [ 313.317104] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.24@tcp is being used as a gateway but routing feature is not turned on [ 313.323494] LNetError: 30287:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 313.940201] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 11:57:07 (1777910227) [ 313.984856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.337192] LNetError: 30286:0:(router.c:1262:lnet_check_routers()) Failed to discover router 192.168.201.24@tcp [ 316.064703] LNetError: 31426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.072538] LNet: Removed LNI 192.168.201.22@tcp [ 316.082716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.106099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 316.116191] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 316.446786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[23-28/2]@tcp [ 316.456685] LNet: 31549:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 316.461788] LNet: 31549:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 316.468660] LNetError: 31436:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.25@tcp has gone from up to down [ 316.472249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[23-28/2]@tcp [ 319.054392] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 11:57:12 (1777910232) [ 319.089983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.102567] LNetError: 32134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 319.111491] LNet: Removed LNI 192.168.201.22@tcp [ 319.123377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.172996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 319.198051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.216948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.252371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.275708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.290298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.311673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 319.342927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.360388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.389873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.429744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.447703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.479571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 319.512234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.528768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.556085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.586763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.603107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.631048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 319.661430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.678843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.707577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.742688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.760722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.786626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 319.818644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.838186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.868959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 319.902224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.920200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.940423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 319.964591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.979700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.007050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 322.716260] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 11:57:16 (1777910236) [ 323.415631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg122-server [ 324.078351] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 11:57:17 (1777910237) [ 324.884063] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg122-server [ 325.650430] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 11:57:19 (1777910239) [ 326.125609] Key type .llcrypt unregistered [ 326.128320] Key type ._llcrypt unregistered [ 328.666023] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 328.721357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 328.728426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 328.733533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 330.999482] Key type ._llcrypt registered [ 331.000727] Key type .llcrypt registered [ 331.080308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 331.086689] alg: No test for adler32 (adler32-zlib) [ 331.895346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.933171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 331.947644] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 332.244275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 332.257499] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 333.221155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 333.233517] LNet: Removed LNI 10.1.2.3@tcp [ 333.288689] LNet: 34261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.292482] LNetError: 34261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 333.305375] LNet: Removed LNI 192.168.201.22@tcp [ 333.610516] Key type .llcrypt unregistered [ 333.612421] Key type ._llcrypt unregistered [ 338.396546] Key type ._llcrypt registered [ 338.398529] Key type .llcrypt registered [ 339.067930] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 11:57:32 (1777910252) [ 339.368278] Key type .llcrypt unregistered [ 339.370524] Key type ._llcrypt unregistered [ 341.970843] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 342.126713] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 342.131213] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 342.136905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 344.399585] Key type ._llcrypt registered [ 344.401047] Key type .llcrypt registered [ 344.486696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 344.494823] alg: No test for adler32 (adler32-zlib) [ 345.302093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.333617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 345.349877] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 345.711220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 345.723098] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 347.019152] LNet: 35754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.022618] LNetError: 35754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 347.026798] LNet: Removed LNI 192.168.201.22@tcp [ 347.038831] LNet: Removed LNI 10.1.2.3@tcp [ 347.329614] Key type .llcrypt unregistered [ 347.330751] Key type ._llcrypt unregistered [ 352.346335] Key type ._llcrypt registered [ 352.348250] Key type .llcrypt registered [ 352.995457] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 11:57:46 (1777910266) [ 353.232521] Key type .llcrypt unregistered [ 353.234058] Key type ._llcrypt unregistered [ 355.711207] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 355.781750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 355.784470] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 355.787137] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 358.053464] Key type ._llcrypt registered [ 358.055284] Key type .llcrypt registered [ 358.175423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 358.185787] alg: No test for adler32 (adler32-zlib) [ 359.049821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.105653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 359.118623] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 360.246756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 360.259243] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 361.273192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 361.285665] LNet: Removed LNI 10.1.2.3@tcp [ 361.298252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 361.321658] LNet: Removed LNI 10.1.2.31@tcp [ 361.330456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 361.345582] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 361.779960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 361.791963] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 362.259428] LNet: 37666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.262883] LNetError: 37666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 362.265333] LNet: Removed LNI 10.1.2.3@tcp [ 362.275169] LNet: Removed LNI 10.1.2.31@tcp [ 362.573573] Key type .llcrypt unregistered [ 362.576757] Key type ._llcrypt unregistered [ 367.717357] Key type ._llcrypt registered [ 367.718987] Key type .llcrypt registered [ 368.444479] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 11:58:01 (1777910281) [ 368.607711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 368.617365] alg: No test for adler32 (adler32-zlib) [ 369.448537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.483126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 369.497067] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 369.908135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.917752] LNet: 38617:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 369.922737] LNetError: 38617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 369.932596] LNet: Removed LNI 192.168.201.22@tcp [ 369.943231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 369.977148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.22@tcp [ 369.993509] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 370.380110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.22@tcp [ 370.401644] LNet: Removed LNI 192.168.201.22@tcp [ 370.432667] LNet: 38752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.730427] Key type .llcrypt unregistered [ 370.732138] Key type ._llcrypt unregistered [ 375.966867] Key type ._llcrypt registered [ 375.968457] Key type .llcrypt registered [ 376.705343] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 11:58:10 (1777910290) [ 377.637434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg122-server [ 378.499950] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 11:58:11 (1777910291) [ 378.798344] Key type .llcrypt unregistered [ 378.803131] Key type ._llcrypt unregistered [ 381.663681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 381.747751] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 381.751029] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 381.754323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 384.062858] Key type ._llcrypt registered [ 384.064236] Key type .llcrypt registered [ 384.166286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 384.175391] alg: No test for adler32 (adler32-zlib) [ 385.015548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.059542] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 385.475614] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 386.680291] LNet: 40588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 386.688099] LNetError: 40588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 386.691698] LNet: Removed LNI 192.168.201.22@tcp1 [ 386.701471] LNet: Removed LNI 10.1.2.3@tcp2 [ 387.086711] Key type .llcrypt unregistered [ 387.094299] Key type ._llcrypt unregistered [ 391.517218] Key type ._llcrypt registered [ 391.518379] Key type .llcrypt registered [ 392.063797] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 11:58:25 (1777910305) [ 392.193676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 392.199537] alg: No test for adler32 (adler32-zlib) [ 393.048182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.075384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 393.090419] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 393.626569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.639192] LNet: 41556:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.643181] LNetError: 41556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 393.661315] LNet: Removed LNI 192.168.201.22@tcp [ 393.672974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.705544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 393.719520] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 396.897460] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 11:58:30 (1777910310) [ 396.963206] LNet: 42139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.968355] LNetError: 42139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 396.982246] LNet: Removed LNI 192.168.201.22@tcp [ 397.286100] Key type .llcrypt unregistered [ 397.288138] Key type ._llcrypt unregistered [ 400.205333] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 400.249737] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 400.255156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 400.261021] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 402.531084] Key type ._llcrypt registered [ 402.532308] Key type .llcrypt registered [ 402.626501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 402.639712] alg: No test for adler32 (adler32-zlib) [ 403.479238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.516906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 403.534220] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 403.962307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 403.976467] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 405.088500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.102195] LNet: 42890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.107402] LNetError: 42890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 405.111194] LNet: Removed LNI 192.168.201.22@tcp [ 405.126274] LNet: Removed LNI 10.1.2.3@tcp [ 405.137650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.167134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 405.180861] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 405.184336] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 405.414655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.427694] LNet: Removed LNI 192.168.201.22@tcp [ 405.437509] LNet: Removed LNI 10.1.2.3@tcp [ 405.448702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.476959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 405.492960] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 405.495723] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 405.704530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 405.718848] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 406.337440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.349272] LNet: 43079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.354371] LNet: 43079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 406.358098] LNetError: 43079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 406.363199] LNetError: 43079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 406.368192] LNet: Removed LNI 192.168.201.22@tcp [ 406.370071] LNet: Removed LNI 10.1.2.3@tcp [ 406.379672] LNet: Removed LNI 192.168.201.22@tcp2 [ 406.397821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.438274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 406.455206] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 406.458101] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 406.461746] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 409.731384] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 11:58:43 (1777910323) [ 409.772261] LNet: 43668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 409.776793] LNetError: 43668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 409.779767] LNetError: 43668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 409.783384] LNet: Removed LNI 192.168.201.22@tcp [ 409.786342] LNet: Removed LNI 10.1.2.3@tcp [ 409.800704] LNet: Removed LNI 192.168.201.22@tcp2 [ 410.100643] Key type .llcrypt unregistered [ 410.102533] Key type ._llcrypt unregistered [ 412.907821] Key type ._llcrypt registered [ 412.908918] Key type .llcrypt registered [ 412.996177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 413.004667] alg: No test for adler32 (adler32-zlib) [ 413.851977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.879339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 413.897780] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 414.296972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.307237] LNet: 44070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.310419] LNetError: 44070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 414.322659] LNet: Removed LNI 192.168.201.22@tcp [ 414.334458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.379358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 414.398707] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 414.644384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.661437] LNet: Removed LNI 192.168.201.22@tcp [ 414.672345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.709644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 414.722425] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 414.957266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.970502] LNet: 44130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.973060] LNet: 44130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 414.977676] LNetError: 44130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 414.983750] LNetError: 44130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 414.998494] LNet: Removed LNI 192.168.201.22@tcp [ 415.014032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.061303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 415.084100] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 415.370442] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 415.407027] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 415.409598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 415.412900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 417.646078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.655858] LNet: 44212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.659492] LNetError: 44212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 417.674824] LNet: Removed LNI 192.168.201.22@tcp [ 417.691543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.728104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 417.748405] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 421.376790] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 11:58:54 (1777910334) [ 421.426934] LNet: 44800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 421.430529] LNetError: 44800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 421.441548] LNet: Removed LNI 192.168.201.22@tcp [ 421.800180] Key type .llcrypt unregistered [ 421.801837] Key type ._llcrypt unregistered [ 424.278266] Key type ._llcrypt registered [ 424.279354] Key type .llcrypt registered [ 424.363897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 424.371357] alg: No test for adler32 (adler32-zlib) [ 425.193558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.220321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 425.232234] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 425.602672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.613176] LNet: 45207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.616430] LNetError: 45207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 425.624471] LNet: Removed LNI 192.168.201.22@tcp [ 425.638970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.672284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 425.689527] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 428.705358] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 11:59:02 (1777910342) [ 428.768175] LNet: 45793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.771644] LNetError: 45793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 428.781891] LNet: Removed LNI 192.168.201.22@tcp [ 429.115640] Key type .llcrypt unregistered [ 429.117616] Key type ._llcrypt unregistered [ 431.453600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 431.497835] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 431.500532] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 431.502941] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 433.754687] Key type ._llcrypt registered [ 433.756353] Key type .llcrypt registered [ 433.854728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 433.863112] alg: No test for adler32 (adler32-zlib) [ 434.693731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.726632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 434.741458] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 435.194287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.204124] LNet: 46247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.207474] LNetError: 46247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.214867] LNet: Removed LNI 192.168.201.22@tcp [ 435.224402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.255399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 435.268661] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 438.574936] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 11:59:12 (1777910352) [ 438.654353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.669077] LNet: 46842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 438.673554] LNetError: 46842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 438.681744] LNet: Removed LNI 192.168.201.22@tcp [ 438.695097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.753647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 441.841949] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 11:59:15 (1777910355) [ 441.934956] LNet: 47422:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.303932] Key type .llcrypt unregistered [ 442.307245] Key type ._llcrypt unregistered [ 445.084889] Key type ._llcrypt registered [ 445.086680] Key type .llcrypt registered [ 445.192433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 445.200477] alg: No test for adler32 (adler32-zlib) [ 446.016299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.045184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 446.065191] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 446.532622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.544568] LNet: 47822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.549183] LNetError: 47822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 446.562622] LNet: Removed LNI 192.168.201.22@tcp [ 446.573630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.611777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 446.627956] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 449.885691] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 11:59:23 (1777910363) [ 449.947200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.962636] LNet: 48409:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.968346] LNetError: 48409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 449.984307] LNet: Removed LNI 192.168.201.22@tcp [ 450.006564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.048247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 453.005941] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 11:59:26 (1777910366) [ 453.798226] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 454.613671] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 11:59:28 (1777910368) [ 455.383214] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 456.247815] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 11:59:29 (1777910369) [ 456.984455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 457.847550] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 11:59:31 (1777910371) [ 458.471980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 459.172217] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 11:59:32 (1777910372) [ 459.845090] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 460.624555] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 11:59:34 (1777910374) [ 460.691383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.704176] LNet: 50464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.715747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.745502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 460.757974] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 461.080535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.22 [ 461.089685] LNet: 50598:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 461.094387] LNet: Removed LNI 192.168.201.22@tcp [ 463.493625] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 11:59:37 (1777910377) [ 463.538635] LNet: 51163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.541256] LNetError: 51163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 463.551424] LNet: Removed LNI 192.168.201.22@tcp [ 463.820651] Key type .llcrypt unregistered [ 463.822636] Key type ._llcrypt unregistered [ 465.881445] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 465.995365] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 466.001078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 466.005439] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 468.349598] Key type ._llcrypt registered [ 468.351381] Key type .llcrypt registered [ 468.440477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 468.448138] alg: No test for adler32 (adler32-zlib) [ 469.455402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 472.556461] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 11:59:45 (1777910385) [ 472.670794] LNet: 52072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 472.674520] LNetError: 52072:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 472.687289] LNet: Removed LNI 10.1.2.3@tcp [ 473.017501] Key type .llcrypt unregistered [ 473.019394] Key type ._llcrypt unregistered [ 476.003591] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 476.133616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 476.136244] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 476.140491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 478.561463] Key type ._llcrypt registered [ 478.562700] Key type .llcrypt registered [ 478.646144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 478.658240] alg: No test for adler32 (adler32-zlib) [ 479.511310] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 482.722495] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 11:59:55 (1777910395) [ 482.785228] LNet: 52953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.793166] LNetError: 52953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 482.810764] LNet: Removed LNI 10.1.2.3@tcp [ 483.168174] Key type .llcrypt unregistered [ 483.170234] Key type ._llcrypt unregistered [ 486.209035] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 486.342529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 486.348695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 486.352622] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 488.792992] Key type ._llcrypt registered [ 488.794663] Key type .llcrypt registered [ 488.894438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.903811] alg: No test for adler32 (adler32-zlib) [ 489.752971] LNet: 53263:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 489.758971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 492.847632] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:00:06 (1777910406) [ 492.956849] LNet: 53855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.959487] LNetError: 53855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 492.971218] LNet: Removed LNI 10.1.2.3@tcp [ 493.258688] Key type .llcrypt unregistered [ 493.260310] Key type ._llcrypt unregistered [ 495.910263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 496.034603] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 496.042631] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 496.048038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 498.406314] Key type ._llcrypt registered [ 498.407987] Key type .llcrypt registered [ 498.508597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 498.515524] alg: No test for adler32 (adler32-zlib) [ 499.374528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.423173] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 503.633305] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:00:17 (1777910417) [ 503.850411] LNet: 55083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.854823] LNetError: 55083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 503.868871] LNet: Removed LNI 10.1.2.3@tcp [ 504.158682] Key type .llcrypt unregistered [ 504.163773] Key type ._llcrypt unregistered [ 506.821996] Key type ._llcrypt registered [ 506.823362] Key type .llcrypt registered [ 506.933695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 506.944123] alg: No test for adler32 (adler32-zlib) [ 507.771773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 507.786616] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 511.908809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 512.255576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 512.271778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 512.289551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 512.300727] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 512.431194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 512.447915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 512.476037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 512.493469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 512.511253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 512.528387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 512.551710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 512.572562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 512.583420] LNet: 55519:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 512.598644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 512.619720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 512.643132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 512.672350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 512.703247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 512.723049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 512.754688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 513.052457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 513.072573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 513.114137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 513.139271] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 513.289735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 513.309421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 513.327714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 513.345207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 513.364294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 513.384639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 513.412468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 513.436844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 513.452697] LNet: 55628:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 513.465431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 513.483697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 513.505652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 513.536442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 513.553292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 513.570150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 513.585481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 513.900233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 513.921077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 513.939747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 513.954148] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 514.035255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 514.057439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 514.080071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 514.099953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 514.127744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 514.151687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 514.177404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 514.194505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 514.211472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 514.231302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 514.251182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 514.267514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 514.281925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 514.309304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 514.325752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 514.574322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 514.601891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 514.624808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 514.635640] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 514.638353] LNet: Skipped 1 previous similar message [ 514.751925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 514.765410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 514.778687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 514.794942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 514.811741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 514.831960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 514.861305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 514.877743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 514.890798] LNet: 55847:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 514.895410] LNet: 55847:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 514.905522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 514.924450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 514.942460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 514.958302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 514.972958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 514.988916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 515.012577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 515.304704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 515.332833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 515.349763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 515.460057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 515.479498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 515.497407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 515.516842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 515.533142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 515.544840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 515.560421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 515.583155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 515.605830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 515.621622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 515.645602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 515.664229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 515.680528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 515.695990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 515.712124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 516.009660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 516.034499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 516.052528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 516.156273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 516.172879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 516.192087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 516.211513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 516.232396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 516.248889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 516.274957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 516.291964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 516.314574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 516.342815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 516.359638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 516.378451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 516.393522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.414462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 516.430520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 516.603434] LNet: 56106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 516.606715] LNetError: 56106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 516.618698] LNet: Removed LNI 192.168.201.22@tcp [ 516.841442] Key type .llcrypt unregistered [ 516.843340] Key type ._llcrypt unregistered [ 524.793277] Key type ._llcrypt registered [ 524.794623] Key type .llcrypt registered [ 525.458320] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:00:38 (1777910438) [ 525.893491] Key type .llcrypt unregistered [ 525.895358] Key type ._llcrypt unregistered [ 528.677290] Key type ._llcrypt registered [ 528.679630] Key type .llcrypt registered [ 528.774186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 528.782900] alg: No test for adler32 (adler32-zlib) [ 529.615783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 529.632103] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 533.193037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 537.389486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.405902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 537.415901] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 538.044119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 538.061857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 1 [ 538.080739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 538.099801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 1 [ 538.122814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_interrupt -r 1 [ 538.141870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 1 [ 538.160458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_interrupt -r 1 [ 538.176874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 1 [ 538.194538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 538.203428] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 538.207611] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 538.305608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 538.323559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 538.338121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 538.355497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 538.369895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 538.389508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 538.407333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 538.428269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 538.438978] LNet: 57568:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 538.449797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 538.469624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 538.494618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 538.514491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 538.535878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 538.552976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 538.568987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 538.779682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 538.798786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 1 [ 538.815952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 538.833459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 1 [ 538.851125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_dropped -r 1 [ 538.870580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 1 [ 538.889313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_dropped -r 1 [ 538.908803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 1 [ 538.929350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 538.940948] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 538.945680] LNet: Skipped 2 previous similar messages [ 539.042284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 539.056230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 539.072771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 539.091281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 539.106331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 539.124984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 539.143342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 539.160631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 539.171773] LNet: 57689:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 539.180152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 539.195300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 539.209727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 539.225972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 539.242929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 539.259036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 539.279914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 539.510643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 539.527201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 1 [ 539.543284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 539.572064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 1 [ 539.595772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_aborted -r 1 [ 539.618513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 1 [ 539.637766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_aborted -r 1 [ 539.661714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 1 [ 539.685230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 539.807840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 539.827766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 539.845438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 539.865641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 539.883719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 539.899192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 539.914496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 539.930858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 539.946639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 539.963703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 539.983345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 540.007109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 540.026272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 540.047107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 540.068520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 540.327742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 540.350208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 1 [ 540.371439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 540.391433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 1 [ 540.410845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_no_route -r 1 [ 540.429435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 1 [ 540.448823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_no_route -r 1 [ 540.466711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 1 [ 540.485504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 540.495234] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 540.498108] LNet: Skipped 6 previous similar messages [ 540.585894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 540.600424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 540.614154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 540.629771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 540.645496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 540.662067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 540.678582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 540.696511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 540.705853] LNet: 57931:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 540.709062] LNet: 57931:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 540.717575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 540.737364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 540.753076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 540.769650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 540.786919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 540.803704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 540.823962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 541.080272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 541.103799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 1 [ 541.127136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 541.145319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 1 [ 541.164838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_timeout -r 1 [ 541.190305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 1 [ 541.210290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_timeout -r 1 [ 541.233490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 1 [ 541.257580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 541.380458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 541.397287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 541.411119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 541.424902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 541.448989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 541.467537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 541.487981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 541.505060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 541.522957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 541.548115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 541.567751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 541.586083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 541.607874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 541.626760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 541.654387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 541.921971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 541.948087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_error -r 1 [ 541.978053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 542.006155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_error -r 1 [ 542.025669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_error -r 1 [ 542.041913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_error -r 1 [ 542.063749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_error -r 1 [ 542.083374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_error -r 1 [ 542.103519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 542.243107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 542.259487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 542.282634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 542.298279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 542.314638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 542.336648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 542.362563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 542.387648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 542.417345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 542.441294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 542.461585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 542.480096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.509254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 542.528851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 542.550159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 545.255730] LNet: 58249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 545.259799] LNetError: 58249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 545.264344] LNet: Removed LNI 192.168.201.22@tcp [ 546.283559] LNet: Removed LNI 192.168.201.22@tcp1 [ 546.517306] Key type .llcrypt unregistered [ 546.519932] Key type ._llcrypt unregistered [ 553.914973] Key type ._llcrypt registered [ 553.916722] Key type .llcrypt registered [ 554.583763] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:01:07 (1777910467) [ 554.967447] Key type .llcrypt unregistered [ 554.970126] Key type ._llcrypt unregistered [ 557.641440] Key type ._llcrypt registered [ 557.642736] Key type .llcrypt registered [ 557.745540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 557.758798] alg: No test for adler32 (adler32-zlib) [ 558.601387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 558.617935] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 562.068640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 562.392367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 562.414463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 562.436410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 562.450343] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 562.549197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 562.562988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 562.577863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 562.593486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 562.607622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 562.625858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 562.648182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 562.671947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 562.679900] LNet: 59508:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 562.688974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 562.707644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 562.729570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 562.750629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 562.769572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 562.787936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 562.808514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 563.038845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 563.058546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 563.082255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 563.093154] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 563.184299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 563.200667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 563.218718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 563.241500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 563.262797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 563.283878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 563.304944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 563.317796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 563.326811] LNet: 59617:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 563.335590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 563.353865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 563.370263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 563.390656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 563.415721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 563.432323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 563.448233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 563.713555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 563.733716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 563.759050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 563.884151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 563.905840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 563.919744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 563.936323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 563.954595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 563.974231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 563.992460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 564.005899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 564.020517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 564.035305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 564.047392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 564.064811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 564.082277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 564.101372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 564.117919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 564.246682] LNet: 59767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.249807] LNetError: 59767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.257823] LNet: Removed LNI 192.168.201.22@tcp [ 564.458510] Key type .llcrypt unregistered [ 564.459760] Key type ._llcrypt unregistered [ 571.678098] Key type ._llcrypt registered [ 571.679141] Key type .llcrypt registered [ 572.421276] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:01:25 (1777910485) [ 572.914322] Key type .llcrypt unregistered [ 572.915506] Key type ._llcrypt unregistered [ 575.583556] Key type ._llcrypt registered [ 575.585205] Key type .llcrypt registered [ 575.680949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 575.691872] alg: No test for adler32 (adler32-zlib) [ 576.507559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 576.524626] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 580.195738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 583.570771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.590546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 583.604327] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 584.250663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 584.267133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 1 [ 584.282986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 584.306017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 1 [ 584.326598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_dropped -r 1 [ 584.345525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 1 [ 584.365501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_dropped -r 1 [ 584.385106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 1 [ 584.403166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 584.412499] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 584.417524] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 584.526320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 584.546976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 584.566446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 584.584247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 584.603411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 584.627567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 584.645818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 584.662745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 584.671836] LNet: 61230:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 584.685312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 584.703248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 584.724397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 584.744969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 584.765407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 584.782403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 584.799967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 584.819989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 585.060190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 585.080166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_error -r 1 [ 585.102229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 585.121605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_error -r 1 [ 585.145807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_error -r 1 [ 585.169497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_error -r 1 [ 585.189311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_error -r 1 [ 585.214983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_error -r 1 [ 585.234726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 585.249035] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 585.251855] LNet: Skipped 2 previous similar messages [ 585.340262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 585.355731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 585.371898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 585.391674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 585.410240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 585.427140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 585.447090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 585.463972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 585.473827] LNet: 61353:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 585.486911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 585.509942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 585.525178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 585.541675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 585.558506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 585.581758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 585.610130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 585.633780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 585.872357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 585.892900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_timeout -r 1 [ 585.931292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 585.957121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_timeout -r 1 [ 585.975879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_timeout -r 1 [ 585.995873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_timeout -r 1 [ 586.017895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_timeout -r 1 [ 586.038732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_timeout -r 1 [ 586.059503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.122@tcp [ 586.152327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 586.170671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 586.187478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 586.203973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 586.222299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 586.245042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 586.285370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 586.311958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 586.331315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 586.349942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 586.369719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 586.384331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 586.399572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 586.420575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 586.435563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 586.451289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 588.962734] LNet: 61554:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.969093] LNetError: 61554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.972459] LNet: Removed LNI 192.168.201.22@tcp [ 588.980638] LNet: Removed LNI 192.168.201.22@tcp1 [ 589.167407] Key type .llcrypt unregistered [ 589.168930] Key type ._llcrypt unregistered [ 593.357525] hrtimer: interrupt took 3372438 ns [ 595.718499] Key type ._llcrypt registered [ 595.720258] Key type .llcrypt registered [ 596.302639] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:01:49 (1777910509) [ 596.523615] Key type .llcrypt unregistered [ 596.525464] Key type ._llcrypt unregistered [ 598.694462] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 598.745064] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 598.746886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 598.748408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 601.032686] Key type ._llcrypt registered [ 601.034504] Key type .llcrypt registered [ 601.116414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 601.124961] alg: No test for adler32 (adler32-zlib) [ 601.936321] LNet: 62627:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 601.943808] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 601.958967] LNet: 62641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.962912] LNetError: 62641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.981974] LNet: Removed LNI 192.168.201.22@tcp [ 602.298522] Key type .llcrypt unregistered [ 602.300734] Key type ._llcrypt unregistered [ 604.679612] Key type ._llcrypt registered [ 604.680960] Key type .llcrypt registered [ 604.754809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 604.766041] alg: No test for adler32 (adler32-zlib) [ 605.580459] LNet: 62900:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 605.587800] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 605.589655] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 605.610868] LNet: 62914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 605.613671] LNetError: 62914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 605.616531] LNet: Removed LNI 192.168.201.22@tcp [ 605.622437] LNet: Removed LNI 10.1.2.3@tcp1 [ 605.919511] Key type .llcrypt unregistered [ 605.920861] Key type ._llcrypt unregistered [ 608.293130] Key type ._llcrypt registered [ 608.294665] Key type .llcrypt registered [ 608.382931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.392119] alg: No test for adler32 (adler32-zlib) [ 609.206038] LNet: 63173:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 609.213160] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 609.215189] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 609.236547] LNet: 63187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 609.239976] LNetError: 63187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.243174] LNet: Removed LNI 192.168.201.22@tcp [ 609.258550] LNet: Removed LNI 10.1.2.3@tcp [ 609.572329] Key type .llcrypt unregistered [ 609.573957] Key type ._llcrypt unregistered [ 611.998399] Key type ._llcrypt registered [ 611.999628] Key type .llcrypt registered [ 612.126904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 612.148796] alg: No test for adler32 (adler32-zlib) [ 613.004827] LNet: 63447:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 613.011255] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 613.013167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 613.030069] LNet: 63461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 613.033494] LNetError: 63461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 613.036743] LNet: Removed LNI 192.168.201.22@tcp [ 613.047698] LNet: Removed LNI 10.1.2.3@tcp [ 613.340358] Key type .llcrypt unregistered [ 613.342300] Key type ._llcrypt unregistered [ 615.914640] Key type ._llcrypt registered [ 615.917111] Key type .llcrypt registered [ 616.012454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 616.018641] alg: No test for adler32 (adler32-zlib) [ 616.839927] LNet: 63724:0:(config.c:1608:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 616.846814] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 616.849831] LNetError: 63724:0:(config.c:1593:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 616.854356] LNetError: Error -100 starting up LNI tcp [ 616.863616] LNet: Removed LNI 192.168.201.22@tcp [ 617.195544] Key type .llcrypt unregistered [ 617.196992] Key type ._llcrypt unregistered [ 622.040301] Key type ._llcrypt registered [ 622.041994] Key type .llcrypt registered [ 622.664166] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:02:16 (1777910536) [ 623.034349] Key type .llcrypt unregistered [ 623.035790] Key type ._llcrypt unregistered [ 625.464454] Key type ._llcrypt registered [ 625.465926] Key type .llcrypt registered [ 625.535016] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 625.542801] alg: No test for adler32 (adler32-zlib) [ 626.361967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 626.375786] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 629.712127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 629.974549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 629.990633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 630.008146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 630.017963] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 630.107936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 630.121906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 630.139893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 630.157392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 630.171914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 630.187746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 630.202467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 630.218634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 630.227186] LNet: 64978:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 630.235661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 630.250648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 630.266511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 630.281276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 630.296564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 630.313162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 630.417039] LNet: 65017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.421113] LNetError: 65017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.429511] LNet: Removed LNI 192.168.201.22@tcp [ 630.604424] Key type .llcrypt unregistered [ 630.606058] Key type ._llcrypt unregistered [ 637.814424] Key type ._llcrypt registered [ 637.816483] Key type .llcrypt registered [ 637.929374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 637.945169] alg: No test for adler32 (adler32-zlib) [ 638.758408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 638.772139] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 642.197223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 645.581502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.597351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 645.608901] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 646.213196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 646.232694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e network_timeout -r 1 [ 646.253636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 646.271224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e network_timeout -r 1 [ 646.290229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e network_timeout -r 1 [ 646.309397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e network_timeout -r 1 [ 646.327141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e network_timeout -r 1 [ 646.345311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e network_timeout -r 1 [ 646.364091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 646.375424] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 646.482372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 646.503794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 646.521924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 646.540779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 646.556610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 646.572968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 646.589868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 646.611304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 646.621036] LNet: 65892:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 646.633717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 646.660441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.678466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 646.697885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 646.718617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 646.738827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 649.047281] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 649.056146] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 649.168001] LNet: 65966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 649.174672] LNetError: 65966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 649.179469] LNet: Removed LNI 192.168.201.22@tcp [ 649.186891] LNet: Removed LNI 192.168.201.22@tcp1 [ 649.365121] Key type .llcrypt unregistered [ 649.366313] Key type ._llcrypt unregistered [ 655.317042] Key type ._llcrypt registered [ 655.318564] Key type .llcrypt registered [ 655.892753] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:02:49 (1777910569) [ 656.086754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 656.093714] alg: No test for adler32 (adler32-zlib) [ 656.917205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.945110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 656.956854] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 657.263155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 657.274043] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 657.635735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 657.663160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 657.683046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 657.702562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 657.720106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 657.735560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 657.750124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 657.758836] LNetError: 67065:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 657.952333] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 657.952381] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 660.003668] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 660.975070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 664.096552] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 664.176840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 664.190116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 664.212719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.222173] LNet: 67173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.225092] LNetError: 67173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.228218] LNet: Removed LNI 192.168.201.22@tcp [ 664.235388] LNet: Removed LNI 192.168.201.22@tcp1 [ 664.245584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.270554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.280779] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 664.600841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 664.610410] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 664.982541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 665.008971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 665.045843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 665.071172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 665.088661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 665.107693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 665.123093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 665.136467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 665.145570] LNetError: 67445:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 667.361150] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 667.364794] LNet: Skipped 1 previous similar message [ 668.370298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 671.456467] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 675.705759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 675.724420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 675.740243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 678.892486] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:03:12 (1777910592) [ 678.943608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 678.959059] LNet: 68158:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 678.962916] LNetError: 68158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 678.966827] LNetError: 68158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 678.969963] LNet: Removed LNI 192.168.201.22@tcp [ 678.983691] LNet: Removed LNI 192.168.201.22@tcp1 [ 679.004773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.040572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 679.055695] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 679.510706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 679.523632] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 679.973276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 680.003679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 680.021308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 680.037687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 680.057793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 680.072648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 680.095777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 0 [ 680.151557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 681.056284] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.22@tcp [ 681.056311] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 681.062107] LNet: Skipped 1 previous similar message [ 695.695512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 695.761242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 695.780237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 695.796391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 500 [ 695.861989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 702.171572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 702.209335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.226040] LNet: 68661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.229232] LNetError: 68661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.232435] LNetError: 68661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 702.235341] LNet: Removed LNI 192.168.201.22@tcp [ 702.244982] LNet: Removed LNI 192.168.201.22@tcp1 [ 702.264749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.310642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.329892] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 702.714521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 702.725713] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 703.179353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 703.198498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 703.234226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 703.264720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 703.288522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 703.306442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 703.324306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 703.340595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.22@tcp --health 0 [ 704.352765] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 704.357667] LNet: Skipped 3 previous similar messages [ 704.471284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 714.982669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 714.999913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 715.017758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 717.529534] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:03:51 (1777910631) [ 717.738468] LNet: 69717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.741477] LNetError: 69717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.744542] LNetError: 69717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 717.747611] LNet: Removed LNI 192.168.201.22@tcp [ 717.757364] LNet: Removed LNI 192.168.201.22@tcp1 [ 717.992702] Key type .llcrypt unregistered [ 717.993875] Key type ._llcrypt unregistered [ 719.982301] Key type ._llcrypt registered [ 719.983641] Key type .llcrypt registered [ 720.052579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 720.058982] alg: No test for adler32 (adler32-zlib) [ 720.872588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 720.884950] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 724.357817] Lustre: DEBUG MARKER: Initial discovery [ 724.373050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 725.181779] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 725.216171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 725.233242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 725.245718] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 725.256273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 725.271634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.122@tcp -r 1 -m GET -l 3 [ 725.288414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.122@tcp -r 1 -m GET -e local_error [ 725.304233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.122@tcp -r 1 -m PUT -l 6 [ 725.843721] Lustre: DEBUG MARKER: Force oleg122-client.virtnet to discover 192.168.201.122@tcp (in background) [ 725.871407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.122@tcp [ 725.984111] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 726.417284] Lustre: DEBUG MARKER: Force oleg122-server to discover 192.168.201.22@tcp [ 727.286402] Lustre: DEBUG MARKER: Wait for 70471 [ 728.928475] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 729.580133] Lustre: DEBUG MARKER: Finished wait on 70471 [ 729.607831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 729.625563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 729.753834] LNet: 70929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.756178] LNetError: 70929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.764502] LNet: Removed LNI 192.168.201.22@tcp [ 729.915647] Key type .llcrypt unregistered [ 729.917297] Key type ._llcrypt unregistered [ 735.844582] Key type ._llcrypt registered [ 735.846113] Key type .llcrypt registered [ 736.410361] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:04:09 (1777910649) [ 736.607599] Key type .llcrypt unregistered [ 736.609597] Key type ._llcrypt unregistered [ 738.989952] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 739.052216] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 739.055225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 739.057854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.311393] Key type ._llcrypt registered [ 741.312457] Key type .llcrypt registered [ 741.401876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.409467] alg: No test for adler32 (adler32-zlib) [ 742.224614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.249194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.262885] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 742.564788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 742.577520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 746.053290] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:04:19 (1777910659) [ 746.103211] LNet: 73006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.105934] LNetError: 73006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.108888] LNet: Removed LNI 192.168.201.22@tcp [ 746.118676] LNet: Removed LNI 10.1.2.3@tcp [ 746.439507] Key type .llcrypt unregistered [ 746.441730] Key type ._llcrypt unregistered [ 749.222336] Key type ._llcrypt registered [ 749.223676] Key type .llcrypt registered [ 749.339181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 749.346785] alg: No test for adler32 (adler32-zlib) [ 750.157433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.184610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.198199] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 750.514923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 750.526381] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 750.989707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.22@tcp [ 752.032631] LNetError: Unexpected error -22 connecting to 192.168.201.22@tcp at host 192.168.201.22:988 [ 753.958334] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:04:27 (1777910667) [ 753.991086] LNet: 74135:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.994739] LNetError: 74135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.997426] LNet: Removed LNI 192.168.201.22@tcp [ 754.004352] LNet: Removed LNI 10.1.2.3@tcp [ 754.267656] Key type .llcrypt unregistered [ 754.269316] Key type ._llcrypt unregistered [ 756.521286] Key type ._llcrypt registered [ 756.522693] Key type .llcrypt registered [ 756.603570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.611528] alg: No test for adler32 (adler32-zlib) [ 757.437206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.468719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 757.481407] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 757.804911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 757.814551] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 758.189574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.22@tcp1 --nid 192.168.201.22@tcp2 [ 759.945923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp1 192.168.201.22@tcp2 [ 759.956326] LNetError: 74977:0:(lib-move.c:2266:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.22@tcp1 to 192.168.201.22@tcp2 - Route table may be misconfigured [ 759.968751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp2 192.168.201.22@tcp1 [ 762.510609] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:04:36 (1777910676) [ 762.544426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.554350] LNet: 75544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.558160] LNetError: 75544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 762.561114] LNet: Removed LNI 192.168.201.22@tcp1 [ 763.623575] LNet: Removed LNI 192.168.201.22@tcp2 [ 763.637954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.660935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.677441] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 763.961795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 763.975841] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 764.299324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 764.314711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.22@tcp [ 764.329768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp1 192.168.201.22@tcp1 [ 764.346482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.22@tcp -d 192.168.201.22@tcp -e network_timeout [ 764.362139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -e network_timeout [ 764.378915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -e network_timeout [ 764.395959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -e network_timeout [ 764.413682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 764.423284] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 764.432167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 764.470698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 764.511107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 764.705115] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 766.979838] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:04:40 (1777910680) [ 767.013335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 767.023653] LNet: 76405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.026662] LNetError: 76405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.030133] LNetError: 76405:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 767.032818] LNet: Removed LNI 192.168.201.22@tcp [ 768.102516] LNet: Removed LNI 192.168.201.22@tcp1 [ 768.111609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.142663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 768.238317] LNet: 76448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.386655] Key type .llcrypt unregistered [ 768.388463] Key type ._llcrypt unregistered [ 772.391634] Key type ._llcrypt registered [ 772.393232] Key type .llcrypt registered [ 772.956360] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:04:46 (1777910686) [ 773.095772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.102222] alg: No test for adler32 (adler32-zlib) [ 773.910726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.449866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 775.219902] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:04:48 (1777910688) [ 775.251068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.261988] LNet: 77539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.276266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.305921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 775.318177] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 775.600072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 775.608621] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 775.966578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 775.983949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp1 [ 776.000763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp1 [ 778.494152] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:04:51 (1777910691) [ 779.160220] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg122-server [ 779.847562] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:04:53 (1777910693) [ 780.607101] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg122-server [ 781.244458] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:04:54 (1777910694) [ 781.975252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg122-server [ 782.633813] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:04:56 (1777910696) [ 783.379072] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg122-server [ 784.056545] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:04:57 (1777910697) [ 784.771652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg122-server [ 785.370098] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:04:58 (1777910698) [ 786.110931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg122-server [ 786.894275] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:05:00 (1777910700) [ 787.633348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg122-server [ 788.324688] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:05:01 (1777910701) [ 789.095676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg122-server [ 789.823050] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:05:03 (1777910703) [ 790.554904] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg122-server [ 791.217452] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:05:04 (1777910704) [ 791.253638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.264922] LNet: 81411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.267309] LNetError: 81411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.269381] LNet: Removed LNI 192.168.201.22@tcp [ 792.294764] LNet: Removed LNI 192.168.201.22@tcp1 [ 792.306679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.339324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.356911] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 792.727054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 792.768463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 792.853548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.862758] LNet: 81564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.866659] LNetError: 81564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.871697] LNetError: 81564:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 792.884687] LNet: Removed LNI 192.168.201.22@tcp [ 792.901420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.932602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 792.945258] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 793.348830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 793.434909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.469761] LNet: Removed LNI 192.168.201.22@tcp [ 794.480516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.508204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 794.520362] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 794.881583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 794.916768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 794.995529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.006056] LNet: 81856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.010438] LNet: 81856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 795.013617] LNetError: 81856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.016625] LNetError: 81856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 795.029380] LNet: Removed LNI 192.168.201.22@tcp [ 795.039269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.064958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 795.079223] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 795.455020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 795.556777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.582702] LNet: Removed LNI 192.168.201.22@tcp [ 796.592836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.619019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.630349] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 796.993096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 797.027785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 797.134482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.150714] LNetError: 82148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.155521] LNetError: 82148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 797.172833] LNet: Removed LNI 192.168.201.22@tcp [ 797.194574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.227321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 797.242068] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 797.597336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 797.721486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.758124] LNet: Removed LNI 192.168.201.22@tcp [ 798.773628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.803476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 798.817842] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 799.144155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 799.178728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 800.338811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.349271] LNet: 82450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.352277] LNet: 82450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 801.382547] LNet: Removed LNI 192.168.201.22@tcp [ 801.393751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.422687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 801.437781] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 801.802646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 801.896528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.916216] LNetError: 82595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.920347] LNetError: 82595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 801.927522] LNet: Removed LNI 192.168.201.22@tcp [ 801.940312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.976658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.989760] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 802.317696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 802.356072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 802.460415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.498812] LNet: Removed LNI 192.168.201.22@tcp [ 803.514852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.552525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 803.572204] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 803.930983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 804.054315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.094361] LNet: Removed LNI 192.168.201.22@tcp [ 805.104757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.130239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.140533] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 805.516708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 805.555442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 805.677419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 806.760179] LNet: Removed LNI 192.168.201.22@tcp [ 806.771574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 806.811729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 806.827470] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 807.215331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 808.451426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.462274] LNet: 83189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.464295] LNet: 83189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 809.514342] LNet: Removed LNI 192.168.201.22@tcp [ 809.528499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.551847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.562354] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 809.876534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 809.900726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.915339] LNet: Removed LNI 192.168.201.22@tcp [ 809.925833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.949691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.959486] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 810.286157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 813.265591] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:05:26 (1777910726) [ 813.301267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.316188] LNetError: 84018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.322296] LNetError: 84018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 813.336206] LNet: Removed LNI 192.168.201.22@tcp [ 813.346789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.374560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 813.390385] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 813.904570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 813.924317] LNet: Removed LNI 192.168.201.22@tcp231 [ 813.932322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 813.963491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 813.977098] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 813.979724] LNet: 84171:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 814.152547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 814.173392] LNet: Removed LNI 192.168.201.22@tcp231 [ 814.180641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 814.192686] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 814.671493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.685346] LNet: Removed LNI 192.168.201.22@tcp231 [ 814.693838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.721151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 814.732204] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/0] [ 814.735442] LNet: 84323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 814.899150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 814.914776] LNet: Removed LNI 192.168.201.22@tcp231 [ 814.921928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 814.931975] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/0] [ 815.381593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.397188] LNet: Removed LNI 192.168.201.22@tcp231 [ 815.406417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.437330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 815.450241] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/60] [ 815.611032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 815.628925] LNet: Removed LNI 192.168.201.22@tcp231 [ 815.637343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 815.653390] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/60] [ 816.154830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.172556] LNet: Removed LNI 192.168.201.22@tcp231 [ 816.180527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.212569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 816.230712] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 816.232729] LNet: 84625:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 816.235538] LNet: 84625:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 816.397994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 816.416594] LNet: Removed LNI 192.168.201.22@tcp231 [ 816.425223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 816.445230] LNet: Added LNI 192.168.201.22@tcp231 [8/256/0/180] [ 819.515398] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:05:33 (1777910733) [ 819.550645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.570303] LNet: Removed LNI 192.168.201.22@tcp231 [ 819.581873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.616382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 819.627921] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 819.913651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 819.929472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 819.949558] LNet: Removed LNI 192.168.201.22@tcp232 [ 819.956886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 819.968627] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 820.270527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 820.287189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 820.308442] LNet: Removed LNI 192.168.201.22@tcp232 [ 820.318591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 820.330865] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 820.617372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 820.635434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 820.653662] LNet: Removed LNI 192.168.201.22@tcp232 [ 820.661334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 820.672357] LNet: Added LNI 192.168.201.22@tcp232 [8/256/0/180] [ 820.981859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 821.001563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 821.020610] LNet: Removed LNI 192.168.201.22@tcp232 [ 821.028515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 823.519336] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:05:37 (1777910737) [ 823.919182] Key type .llcrypt unregistered [ 823.920674] Key type ._llcrypt unregistered [ 826.094364] Key type ._llcrypt registered [ 826.095703] Key type .llcrypt registered [ 826.180285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.187815] alg: No test for adler32 (adler32-zlib) [ 827.001974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 827.014774] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 830.191756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 833.843031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 833.860480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 833.872182] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 834.514333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_interrupt -r 2 [ 834.541958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 2 [ 834.559771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_interrupt -r 2 [ 834.578942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 2 [ 834.596200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_interrupt -r 2 [ 834.617657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_interrupt -r 2 [ 834.635305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_interrupt -r 2 [ 834.653163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_interrupt -r 2 [ 834.671538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 834.681511] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 834.684594] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 834.783691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 834.801734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 834.821447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 834.837789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 834.853887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 834.870996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 834.887410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 834.905684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 834.914081] LNet: 87121:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 834.923732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 834.939311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 834.955504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 834.976199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 834.998585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 835.024964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 835.287665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_dropped -r 2 [ 835.306424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 2 [ 835.324991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_dropped -r 2 [ 835.348306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 2 [ 835.378945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_dropped -r 2 [ 835.403730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_dropped -r 2 [ 835.426994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_dropped -r 2 [ 835.446925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_dropped -r 2 [ 835.470628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 835.487562] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 835.628124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 835.645398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 835.662760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 835.679507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 835.695424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 835.717675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 835.735837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 835.757060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 835.770138] LNet: 87241:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 835.782447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 835.809781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 835.825681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 835.845216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 835.863236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 835.882376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 836.203874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_aborted -r 2 [ 836.225537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 2 [ 836.246532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_aborted -r 2 [ 836.265673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 2 [ 836.285535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_aborted -r 2 [ 836.304613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_aborted -r 2 [ 836.331552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_aborted -r 2 [ 836.361681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_aborted -r 2 [ 836.403737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 836.599950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 836.626770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 836.650583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 836.673294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 836.692749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 836.713185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 836.732125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 836.753239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 836.782313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 836.807784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 836.831579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 836.852717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 836.868889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 836.885347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 837.134572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_no_route -r 2 [ 837.151133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 2 [ 837.176539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_no_route -r 2 [ 837.194728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 2 [ 837.210635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_no_route -r 2 [ 837.230411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_no_route -r 2 [ 837.250594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_no_route -r 2 [ 837.285132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_no_route -r 2 [ 837.309605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 837.322127] LNet: There was an unexpected network error while writing to 192.168.201.122: rc = -22 [ 837.325509] LNet: Skipped 2 previous similar messages [ 837.500923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.520502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.544348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.571365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.599313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.619628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.639284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.660310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.679823] LNet: 87481:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 837.690841] LNet: 87481:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 837.706978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.735163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.757666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.781624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.804139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 837.822795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.115751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e local_timeout -r 2 [ 838.132954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 2 [ 838.149985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e local_timeout -r 2 [ 838.172759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 2 [ 838.194730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e local_timeout -r 2 [ 838.215817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e local_timeout -r 2 [ 838.241351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e local_timeout -r 2 [ 838.264941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e local_timeout -r 2 [ 838.288833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 838.431140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 838.452870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 838.469205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 838.484942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 838.507343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 838.524341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 838.542552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 838.559756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 838.573768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 838.594756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 838.612749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 838.634853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 838.662275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.694305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.975923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp -m GET -e remote_dropped -r 2 [ 839.006623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 2 [ 839.028905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp -m GET -e remote_dropped -r 2 [ 839.056631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 2 [ 839.086893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp -m GET -e remote_dropped -r 2 [ 839.107899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.122@tcp1 -m GET -e remote_dropped -r 2 [ 839.131590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp -m GET -e remote_dropped -r 2 [ 839.153827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.22@tcp1 -d 192.168.201.22@tcp1 -m GET -e remote_dropped -r 2 [ 839.171886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.22@tcp 192.168.201.122@tcp [ 839.185915] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 839.320567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.358553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.390950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.415240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.432233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.452652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.476855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.490624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.507139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.523045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.538452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.553377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.572118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.586732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 842.279478] LNet: 87795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.283573] LNetError: 87795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.287419] LNet: Removed LNI 192.168.201.22@tcp [ 842.302077] LNet: Removed LNI 192.168.201.22@tcp1 [ 842.486677] Key type .llcrypt unregistered [ 842.488338] Key type ._llcrypt unregistered [ 849.332570] Key type ._llcrypt registered [ 849.334738] Key type .llcrypt registered [ 849.915716] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:06:03 (1777910763) [ 850.109266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 850.118174] alg: No test for adler32 (adler32-zlib) [ 850.969222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.003538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 851.017637] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 851.490924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 851.514644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.22@tcp [ 851.530411] LNetError: 88772:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 851.578931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.22@tcp [ 851.628624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 854.954834] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:06:08 (1777910768) [ 855.026272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.039110] LNet: 89359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.045129] LNetError: 89359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.056718] LNet: Removed LNI 192.168.201.22@tcp [ 855.070509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.102945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.123074] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 855.516783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 855.529955] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 856.068099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.22@tcp [ 856.079674] LNetError: 89627:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.201.22@tcp to recovery [ 856.098401] LNet: 1 local NIs in recovery (showing 1): 192.168.201.22@tcp [ 856.099617] LNet: There was an unexpected network error while writing to 192.168.201.22: rc = -22 [ 856.147794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.22@tcp [ 861.340601] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:06:14 (1777910774) [ 862.198888] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg122-server [ 862.984252] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:06:16 (1777910776) [ 863.765706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg122-server [ 864.514261] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:06:18 (1777910778) [ 865.428622] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg122-server [ 866.286679] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:06:19 (1777910779) [ 866.352825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.371134] LNet: 91246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.378699] LNetError: 91246:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.382848] LNet: Removed LNI 192.168.201.22@tcp [ 866.389754] LNet: Removed LNI 192.168.201.22@tcp1 [ 866.404362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.439384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.449890] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 866.799747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 866.835756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 866.866926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 866.911751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 866.932060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 866.951696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 869.749854] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:06:23 (1777910783) [ 869.788826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.798948] LNet: 91967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.804484] LNetError: 91967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.806626] LNetError: 91967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 869.813637] LNet: Removed LNI 192.168.201.22@tcp [ 869.823827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.849668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.861963] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 872.741615] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:06:26 (1777910786) [ 872.799956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.812913] LNet: 92670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.818454] LNetError: 92670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.824642] LNet: Removed LNI 192.168.201.22@tcp [ 872.838876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.867735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 872.879414] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 875.594887] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:06:29 (1777910789) [ 876.152133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 876.803493] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:06:30 (1777910790) [ 876.966939] LNet: 93708:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.970472] LNetError: 93708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 876.980539] LNet: Removed LNI 192.168.201.22@tcp [ 877.242845] Key type .llcrypt unregistered [ 877.245132] Key type ._llcrypt unregistered [ 879.482175] Key type ._llcrypt registered [ 879.483446] Key type .llcrypt registered [ 879.559083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 879.567624] alg: No test for adler32 (adler32-zlib) [ 880.380827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 880.394741] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 883.663917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 885.276443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.122@tcp [ 885.403231] LNet: 94124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.408440] LNetError: 94124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.421416] LNet: Removed LNI 192.168.201.22@tcp [ 885.592757] Key type .llcrypt unregistered [ 885.594340] Key type ._llcrypt unregistered [ 889.757521] Key type ._llcrypt registered [ 889.759056] Key type .llcrypt registered [ 890.329822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 890.950036] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 891.655067] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:06:45 (1777910805) [ 891.791612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 891.797610] alg: No test for adler32 (adler32-zlib) [ 892.646514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.690977] LNet: 95209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.979527] Key type .llcrypt unregistered [ 892.980876] Key type ._llcrypt unregistered [ 895.246396] Key type ._llcrypt registered [ 895.247727] Key type .llcrypt registered [ 895.332554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 895.341095] alg: No test for adler32 (adler32-zlib) [ 896.172985] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 896.174600] LNet: 95464:0:(config.c:1245:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 896.180079] LNet: 95464:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 896.193429] LNetError: 95476:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 896.198978] LNetError: 95476:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.201.26@tcp has gone from up to down [ 896.419171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.488162] LNetError: 95476:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) route to o2ib through 192.168.201.23@tcp has gone from up to down [ 899.492841] LNetError: 95476:0:(lib-lnet.h:1300:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 899.496368] LNet: 95522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 899.500192] LNetError: 95522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.509342] LNet: Removed LNI 192.168.201.22@tcp [ 899.520083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.545835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 899.557671] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 899.561208] LNet: 95530:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 899.566527] LNet: 95530:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 899.570821] LNetError: 95530:0:(peer.c:367:lnet_peer_ni_del_locked()) Peer NI 192.168.201.23@tcp is a gateway. Can not delete it [ 899.577617] LNetError: 95530:0:(peer.c:2167:lnet_del_peer_ni()) 192.168.201.23@tcp is a router. Can not be deleted [ 899.586352] LNetError: 95527:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.26@tcp is being used as a gateway but routing feature is not turned on [ 899.594904] LNetError: 95527:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 899.598820] LNet: 95530:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.273377] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:06:55 (1777910815) [ 903.013435] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg122-server [ 903.874081] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:06:57 (1777910817) [ 904.664498] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg122-server [ 905.447786] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:06:58 (1777910818) [ 906.086431] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 907.768154] Key type lgssc registered [ 908.185171] Lustre: Echo OBD driver; http://www.lustre.org/ [ 916.807432] Key type lgssc unregistered [ 916.944043] LNet: 97948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.946918] LNetError: 97948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.955509] LNet: Removed LNI 192.168.201.22@tcp [ 917.173807] Key type .llcrypt unregistered [ 917.175075] Key type ._llcrypt unregistered [ 922.669928] Key type ._llcrypt registered [ 922.671264] Key type .llcrypt registered [ 923.195136] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:07:16 (1777910836) [ 923.321390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 923.328941] alg: No test for adler32 (adler32-zlib) [ 924.135136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 924.164835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 924.185131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 924.208965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 924.232853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 926.662640] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:07:20 (1777910840) [ 926.703492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 926.718519] LNet: 99346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.731320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 926.764887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 926.776667] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 927.158490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.167849] LNetError: 99496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 927.181355] LNet: Removed LNI 192.168.201.22@tcp [ 927.189520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.212737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 927.227337] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 927.416044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 927.425473] LNet: 99524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 927.428811] LNet: 99524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 927.436343] LNet: Removed LNI 192.168.201.22@tcp [ 927.444898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.473587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 927.484891] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 930.140455] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:07:23 (1777910843) [ 930.178953] LNet: 100106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.181866] LNetError: 100106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.184636] LNetError: 100106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 930.191787] LNet: Removed LNI 192.168.201.22@tcp [ 930.458581] Key type .llcrypt unregistered [ 930.460296] Key type ._llcrypt unregistered [ 932.679804] Key type ._llcrypt registered [ 932.681642] Key type .llcrypt registered [ 932.759456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.766616] alg: No test for adler32 (adler32-zlib) [ 933.573051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.606323] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 933.608618] LNet: *** cfs_fail_loc=e003, val=100*** [ 933.948900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 933.962105] Ping reply received with mapping: [ 933.964625] found metadata reply NI[192.168.201.22@tcp] [ 933.966602] 16 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 933.968294] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 933.970505] 5a 5a 5a 5a 5a 5a 5a 5a [ 934.025858] LNet: 100511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.030140] LNetError: 100511:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.037527] LNet: Removed LNI 192.168.201.22@tcp [ 934.293512] Key type .llcrypt unregistered [ 934.295027] Key type ._llcrypt unregistered [ 938.685869] Key type ._llcrypt registered [ 938.687427] Key type .llcrypt registered [ 939.295277] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:07:32 (1777910852) [ 939.480419] Key type .llcrypt unregistered [ 939.481972] Key type ._llcrypt unregistered [ 941.806908] Key type ._llcrypt registered [ 941.808286] Key type .llcrypt registered [ 941.895463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 941.902657] alg: No test for adler32 (adler32-zlib) [ 942.792528] LNetError: 101554:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 942.937366] Key type .llcrypt unregistered [ 942.938663] Key type ._llcrypt unregistered [ 945.090784] Key type ._llcrypt registered [ 945.092280] Key type .llcrypt registered [ 945.695651] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:07:39 (1777910859) [ 945.950410] Key type .llcrypt unregistered [ 945.952066] Key type ._llcrypt unregistered [ 948.291681] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.300974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 950.542836] Key type ._llcrypt registered [ 950.544412] Key type .llcrypt registered [ 950.619640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 950.630039] alg: No test for adler32 (adler32-zlib) [ 951.445537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 951.480095] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 956.012821] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:07:49 (1777910869) [ 956.064555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.076948] LNet: 103328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.080490] LNetError: 103328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 956.088579] LNet: Removed LNI 10.1.2.3@tcp [ 956.107612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.434499] Key type .llcrypt unregistered [ 956.436090] Key type ._llcrypt unregistered [ 958.925350] Key type ._llcrypt registered [ 958.926625] Key type .llcrypt registered [ 959.006237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.013685] alg: No test for adler32 (adler32-zlib) [ 959.829261] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 959.861635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 959.930569] LNet: 103620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.934685] LNetError: 103620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 959.949192] LNet: Removed LNI 192.168.201.22@tcp [ 960.257367] Key type .llcrypt unregistered [ 960.258940] Key type ._llcrypt unregistered [ 964.721365] Key type ._llcrypt registered [ 964.722734] Key type .llcrypt registered [ 965.323981] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:07:58 (1777910878) [ 965.570920] Key type .llcrypt unregistered [ 965.574286] Key type ._llcrypt unregistered [ 968.172677] Key type ._llcrypt registered [ 968.174779] Key type .llcrypt registered [ 968.285877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.294294] alg: No test for adler32 (adler32-zlib) [ 971.534135] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:08:05 (1777910885) [ 971.576142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 971.596257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 971.632452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 971.648309] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 971.953820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 971.968775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 971.986451] LNet: Removed LNI 192.168.201.22@tcp [ 972.080839] LNet: 105362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.238346] Key type .llcrypt unregistered [ 972.239875] Key type ._llcrypt unregistered [ 976.361803] Key type ._llcrypt registered [ 976.363462] Key type .llcrypt registered [ 976.951331] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:08:10 (1777910890) [ 977.098113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 977.104039] alg: No test for adler32 (adler32-zlib) [ 977.919387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.947336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 977.960807] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 978.273377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 978.287852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.22@tcp [ 978.295914] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 978.297914] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 978.299300] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 978, 0, 0 [ 978.301258] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 978.302651] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 978.304580] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 978.306064] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 978, 0, 0 [ 978.308101] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 978.309365] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 978.311579] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 978.313626] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 978, 0, 1028 [ 978.317120] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 978.318264] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 978.319970] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 978.322359] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 978, 0, 1028 [ 978.325783] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 981.048837] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:08:14 (1777910894) [ 981.328671] LNet: 106901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 981.338244] LNetError: 106901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 981.351779] LNet: Removed LNI 192.168.201.22@tcp [ 981.667320] Key type .llcrypt unregistered [ 981.669895] Key type ._llcrypt unregistered [ 984.848742] Key type ._llcrypt registered [ 984.849836] Key type .llcrypt registered [ 984.978854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.987924] alg: No test for adler32 (adler32-zlib) [ 985.810089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 985.824975] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 989.582051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 993.547407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 993.564870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 993.576521] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 994.017711] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.080117] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.082214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.084482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.243678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 996.253830] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1001.034887] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:08:34 (1777910914) [ 1001.104559] LNet: 108406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.108037] LNetError: 108406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.112373] LNet: Removed LNI 192.168.201.22@tcp [ 1002.144191] LNet: Removed LNI 192.168.201.22@tcp1 [ 1002.145748] LNetError: 108406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.148617] LNetError: 108406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1002.158416] LNet: Removed LNI 10.1.2.3@tcp99 [ 1002.415580] Key type .llcrypt unregistered [ 1002.417270] Key type ._llcrypt unregistered [ 1004.904274] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1004.907498] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.171634] Key type ._llcrypt registered [ 1007.172895] Key type .llcrypt registered [ 1007.261383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.270863] alg: No test for adler32 (adler32-zlib) [ 1008.099968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.128459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1008.139792] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1008.484917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1008.496548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.508549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1009.529504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1009.549323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1009.561355] LNetError: 109149:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1009.577973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1009.598761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1009.783881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1009.824850] LNet: 109169:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.828854] LNetError: 109169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.834634] LNet: Removed LNI 192.168.201.22@tcp [ 1009.841959] LNet: Removed LNI 10.1.2.3@tcp [ 1010.185586] Key type .llcrypt unregistered [ 1010.187796] Key type ._llcrypt unregistered [ 1015.199310] Key type ._llcrypt registered [ 1015.200913] Key type .llcrypt registered [ 1015.883546] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:08:49 (1777910929) [ 1016.082765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1016.090107] alg: No test for adler32 (adler32-zlib) [ 1016.923841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.955214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1016.966216] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1017.376704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.22@tcp [ 1017.413347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg122-client@tcp [ 1020.043514] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:08:53 (1777910933) [ 1020.131421] LNet: 110701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1020.134063] LNetError: 110701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.145931] LNet: Removed LNI 192.168.201.22@tcp [ 1020.424498] Key type .llcrypt unregistered [ 1020.426624] Key type ._llcrypt unregistered [ 1022.584033] Key type ._llcrypt registered [ 1022.585038] Key type .llcrypt registered [ 1022.658370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.665813] alg: No test for adler32 (adler32-zlib) [ 1023.471712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1023.484171] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1023.497843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1025.794531] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:08:59 (1777910939) [ 1026.327302] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1026.994227] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:09:00 (1777910940) [ 1027.597186] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1028.256751] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:09:01 (1777910941) [ 1028.296139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1028.311919] LNet: 112114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.314777] LNetError: 112114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.328867] LNet: Removed LNI 192.168.201.22@tcp [ 1028.343612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1028.379364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1028.395120] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1028.731462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1028.758597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1031.717677] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:09:05 (1777910945) [ 1031.791259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1031.808447] LNet: 112826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1031.814396] LNetError: 112826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1031.825914] LNet: Removed LNI 192.168.201.22@tcp [ 1031.837399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.871910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1031.897659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1032.223492] Key type .llcrypt unregistered [ 1032.225063] Key type ._llcrypt unregistered [ 1037.553995] Key type ._llcrypt registered [ 1037.555451] Key type .llcrypt registered [ 1038.170517] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:09:11 (1777910951) [ 1038.393307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1038.401122] alg: No test for adler32 (adler32-zlib) [ 1039.229647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1039.261137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1039.276088] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1039.635339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1039.670223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1042.823534] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:09:16 (1777910956) [ 1042.913174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.926692] LNet: 114381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.930587] LNetError: 114381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.950662] LNet: Removed LNI 192.168.201.22@tcp [ 1042.970551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1043.004111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1043.023767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1045.848862] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:09:19 (1777910959) [ 1045.902390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1045.915366] LNet: 114961:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.929156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.960385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1046.138633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1046.153474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1046.179890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1046.335322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1046.353862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1046.385514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.400852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.436687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1046.597612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1046.614300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1046.646716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1046.806215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1046.820617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1049.838942] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:09:23 (1777910963) [ 1049.872979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1049.882375] LNet: 115588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1049.885394] LNet: 115588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1049.897268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1049.929110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1049.940374] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1050.291071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1050.306994] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1050.688625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1050.716781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1050.828167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1050.837936] LNetError: 115874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1050.840835] LNet: Removed LNI 192.168.201.22@tcp [ 1050.849348] LNet: Removed LNI 192.168.201.22@tcp2 [ 1050.859808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1050.898652] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1050.902480] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1050.905699] LNet: 115880:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1051.115465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1051.219578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1052.256233] LNet: Removed LNI 192.168.201.22@tcp [ 1052.257812] LNetError: 115917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.260714] LNetError: 115917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1052.270365] LNet: Removed LNI 192.168.201.22@tcp2 [ 1052.281354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.311783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1052.329440] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1052.689127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1052.701550] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1053.156354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1053.183876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1053.222712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1053.331662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1053.349289] LNet: 116201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.352833] LNet: 116201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1054.368223] LNet: Removed LNI 192.168.201.22@tcp [ 1054.372516] LNetError: 116201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.377752] LNetError: 116201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1054.394819] LNet: Removed LNI 192.168.201.22@tcp2 [ 1054.413177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.457827] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1054.461222] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1054.464415] LNet: 116207:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1054.708371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1054.840676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.856624] LNet: Removed LNI 192.168.201.22@tcp [ 1055.911211] LNet: Removed LNI 192.168.201.22@tcp2 [ 1055.932261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.968341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1055.988668] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1056.400057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1056.414634] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1056.863682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1056.900224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1057.000936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.016391] LNet: Removed LNI 192.168.201.22@tcp [ 1058.026173] LNet: Removed LNI 192.168.201.22@tcp2 [ 1058.043917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.100139] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1058.105087] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1058.108916] LNet: 116530:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1058.312733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1058.400418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1058.412560] LNet: 116567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.414837] LNet: 116567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1058.417608] LNetError: 116567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.420311] LNetError: 116567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1058.422653] LNet: Removed LNI 192.168.201.22@tcp [ 1058.433371] LNet: Removed LNI 192.168.201.22@tcp2 [ 1058.443498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.475526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1058.488244] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1058.810900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1058.820714] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1059.216637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1059.237862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1059.266508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1059.358761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1060.384244] LNet: Removed LNI 192.168.201.22@tcp [ 1061.414764] LNet: Removed LNI 192.168.201.22@tcp2 [ 1061.425931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.453237] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1061.456038] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1061.458679] LNet: 116857:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.663829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1061.752476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.763536] LNet: Removed LNI 192.168.201.22@tcp [ 1062.822787] LNet: Removed LNI 192.168.201.22@tcp2 [ 1062.833954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.868476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.883772] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1063.231189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1063.249951] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1063.640647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1063.657543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1063.685775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1063.836964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1064.864177] LNet: Removed LNI 192.168.201.22@tcp [ 1064.874804] LNet: Removed LNI 192.168.201.22@tcp2 [ 1064.888373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1064.917182] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1064.920347] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1065.102715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1065.261200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.336136] LNet: Removed LNI 192.168.201.22@tcp [ 1066.344733] LNet: Removed LNI 192.168.201.22@tcp2 [ 1066.355258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.381115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1066.393725] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1066.766929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1066.777760] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1067.204357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1067.222417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1067.235406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1067.258750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1067.399664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1067.414949] LNet: 117550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.417686] LNet: 117550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1067.420623] LNetError: 117550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.423766] LNetError: 117550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1067.427042] LNet: Removed LNI 192.168.201.22@tcp [ 1068.454341] LNet: Removed LNI 192.168.201.22@tcp2 [ 1068.465779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.501796] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1068.507377] LNet: Added LNI 192.168.201.22@tcp2 [8/256/0/180] [ 1068.509990] LNet: 117556:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.514970] LNet: 117556:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1068.722486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1071.435213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1072.036234] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:09:45 (1777910985) [ 1072.085349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1072.096594] LNet: Removed LNI 192.168.201.22@tcp [ 1073.126878] LNet: Removed LNI 192.168.201.22@tcp2 [ 1073.141062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.255065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1073.268338] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1073.573946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1073.584158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1073.978839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1073.989034] LNet: Removed LNI 192.168.201.22@tcp [ 1073.998242] LNet: Removed LNI 10.1.2.3@tcp [ 1074.010168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1074.043196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1074.059804] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1074.063149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1074.260976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1074.276896] LNet: Removed LNI 192.168.201.22@tcp [ 1074.289487] LNet: Removed LNI 10.1.2.3@tcp [ 1076.805048] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:09:50 (1777910990) [ 1076.869595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1076.898273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.952600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.038756] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1077.040722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1077.042599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.215107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1079.231661] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1079.585654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1079.596195] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1080.548252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.22@tcp [ 1080.567816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1080.606376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.22@tcp -m PUT -e local_timeout -r 1 [ 1080.643565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1080.670682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1080.675698] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1080.775248] LNetError: 119254:0:(lib-move.c:4969:lnet_parse()) 192.168.201.22@tcp, src 192.168.201.22@tcp: Dropping PUT (error -108 looking up sender) [ 1082.848184] LNet: Removed LNI 192.168.201.22@tcp [ 1082.855463] LNet: Removed LNI 10.1.2.3@tcp [ 1083.006884] Key type .llcrypt unregistered [ 1083.008266] Key type ._llcrypt unregistered [ 1086.942163] Key type ._llcrypt registered [ 1086.943338] Key type .llcrypt registered [ 1087.447214] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:10:01 (1777911001) [ 1087.622553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1087.629395] alg: No test for adler32 (adler32-zlib) [ 1088.437107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.623664] LNet: 120563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.868469] Key type .llcrypt unregistered [ 1088.869848] Key type ._llcrypt unregistered [ 1090.997913] Key type ._llcrypt registered [ 1090.999089] Key type .llcrypt registered [ 1091.077871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.084758] alg: No test for adler32 (adler32-zlib) [ 1091.897958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1091.911141] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1095.051127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1095.116953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1095.352319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.122@tcp [ 1095.454508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.468663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1095.483823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.498918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.513293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.529771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.548642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.566223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.575548] LNet: 121011:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.587937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.610750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.628356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.643431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.656683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.672607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.913787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1096.019773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.032295] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.122@tcp [ 1096.035891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1096.050322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.068255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.084467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.101422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.116031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.132671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.141367] LNet: 121128:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1096.151119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.167614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.189093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.204935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1096.223864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1096.239843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1096.490635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.122@tcp [ 1096.597790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.611908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1096.628640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.646210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.662276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.680415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.696525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.711911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.728560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.744620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.760554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.774929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1096.788727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1096.809829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.058247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1097.158583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1097.173836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1097.189880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1097.205574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1097.223917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1097.239248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1097.258840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1097.277392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1097.287659] LNet: 121362:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1097.291030] LNet: 121362:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1097.306894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1097.322460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1097.337346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1097.352883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.369197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.395847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.532387] LNet: 121408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.535780] LNetError: 121408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.546527] LNet: Removed LNI 192.168.201.22@tcp [ 1097.712891] Key type .llcrypt unregistered [ 1097.715068] Key type ._llcrypt unregistered [ 1104.057376] Key type ._llcrypt registered [ 1104.058605] Key type .llcrypt registered [ 1104.637712] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:10:18 (1777911018) [ 1104.787689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.794797] alg: No test for adler32 (adler32-zlib) [ 1105.630228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.817814] LNet: 122294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1106.101328] Key type .llcrypt unregistered [ 1106.103065] Key type ._llcrypt unregistered [ 1108.440312] Key type ._llcrypt registered [ 1108.441409] Key type .llcrypt registered [ 1108.530660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.538265] alg: No test for adler32 (adler32-zlib) [ 1109.373960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1109.389102] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1112.489092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.122@tcp [ 1115.365208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1115.383291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1115.394277] LNet: Added LNI 192.168.201.22@tcp1 [8/256/0/180] [ 1115.732736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1115.948993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.122@tcp [ 1116.036233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.049779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.064028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.078032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.091787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.105441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.120928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.135906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.143848] LNet: 122943:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.153067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.165221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.179299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.193747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.208368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.222195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.463555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1116.554977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.570236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.576192] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.122@tcp, 192.168.201.122@tcp1 [ 1116.583874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.597389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.611418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.625839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.641742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.655336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.663641] LNet: 123060:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.674514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.690713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.706048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.720894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.734862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.749308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.980323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.122@tcp [ 1117.081223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.095168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1117.109568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.123579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.137634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.152139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.166133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.178995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.200292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.216378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.232396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.248224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.264434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.281023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.524324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1117.616345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.631851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1117.648905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1117.664409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.684369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.704033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.720399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.735745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.745261] LNet: 123295:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.748532] LNet: 123295:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1117.758277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.773650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.787578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.800879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.814994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.830310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.199705] LNet: 123375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.203695] LNetError: 123375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.207547] LNet: Removed LNI 192.168.201.22@tcp [ 1120.214566] LNet: Removed LNI 192.168.201.22@tcp1 [ 1120.364711] Key type .llcrypt unregistered [ 1120.366182] Key type ._llcrypt unregistered [ 1126.894036] Key type ._llcrypt registered [ 1126.895470] Key type .llcrypt registered [ 1127.506314] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:10:40 (1777911040) [ 1130.340160] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:10:43 (1777911043) [ 1133.444320] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:10:46 (1777911046) [ 1136.301324] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:10:49 (1777911049) [ 1139.576735] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:10:52 (1777911052) [ 1142.486807] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:10:56 (1777911056) [ 1145.492886] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:10:59 (1777911059) [ 1148.197080] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:11:01 (1777911061) [ 1151.289252] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:11:04 (1777911064) [ 1154.245352] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:11:07 (1777911067) [ 1157.043805] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:11:10 (1777911070) [ 1159.921306] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:11:13 (1777911073) [ 1162.475134] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:11:16 (1777911076) [ 1165.099685] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:11:18 (1777911078) [ 1168.095411] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:11:21 (1777911081) [ 1171.049491] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:11:24 (1777911084) [ 1174.153108] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:11:27 (1777911087) [ 1176.994322] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:11:30 (1777911090) [ 1177.137795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1177.144506] alg: No test for adler32 (adler32-zlib) [ 1177.975165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.039733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1178.057844] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1178.652655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1178.666090] LNet: 134070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.669979] LNetError: 134070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.679380] LNet: Removed LNI 192.168.201.22@tcp [ 1178.690859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.735327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1178.748197] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1182.229691] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:11:35 (1777911095) [ 1185.525304] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:11:38 (1777911098) [ 1189.050550] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:11:42 (1777911102) [ 1192.441878] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:11:45 (1777911105) [ 1195.268845] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:11:48 (1777911108) [ 1198.407982] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:11:51 (1777911111) [ 1201.751792] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:11:55 (1777911115) [ 1205.243705] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:11:58 (1777911118) [ 1208.853600] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:12:02 (1777911122) [ 1212.520027] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:12:05 (1777911125) [ 1215.603601] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:12:09 (1777911129) [ 1218.585670] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:12:12 (1777911132) [ 1221.452371] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1071 sec ========== 12:12:14 (1777911134) [ 1221.555290] LNet: 141526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.557823] LNetError: 141526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.565625] LNet: Removed LNI 192.168.201.22@tcp [ 1221.830846] Key type .llcrypt unregistered [ 1221.833192] Key type ._llcrypt unregistered [ 1227.328668] Key type ._llcrypt registered [ 1227.330196] Key type .llcrypt registered [ 1227.471899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.481454] alg: No test for adler32 (adler32-zlib) [ 1228.430927] Lustre: Lustre: Build Version: 2.17.52_127_gcee2859 [ 1228.575117] LNet: Added LNI 192.168.201.22@tcp [8/256/0/180] [ 1230.192124] Key type lgssc registered [ 1230.856448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1255.072659] Lustre: Mounted lustre-client [ 1257.589319] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1260.358733] LustreError: 143962:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9656124d8000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1260.379073] Lustre: Unmounted lustre-client [ 1272.649847] Key type lgssc unregistered [ 1272.778660] LNet: 144441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.781853] LNetError: 144441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.790904] LNet: Removed LNI 192.168.201.22@tcp [ 1273.051104] Key type .llcrypt unregistered [ 1273.052460] Key type ._llcrypt unregistered