[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 434801695 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002407] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007262] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010141] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013069] *** VALIDATE selinux *** [ 0.021421] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026242] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027157] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] *** VALIDATE tmpfs *** [ 0.031038] *** VALIDATE proc *** [ 0.032200] *** VALIDATE cgroup *** [ 0.033009] *** VALIDATE cgroup2 *** [ 0.035179] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037000] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.040076] debug: unmapping init [mem 0xffffffff94059000-0xffffffff94060fff] [ 0.042192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043656] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047011] ... value mask: 0000ffffffffffff [ 0.048012] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050012] ... event mask: 000000070000000f [ 0.051326] rcu: Hierarchical SRCU implementation. [ 0.053540] smp: Bringing up secondary CPUs ... [ 0.054511] x86: Booting SMP configuration: [ 0.055019] .... node #0, CPUs: #1 #2 #3 [ 0.059012] smp: Brought up 1 node, 4 CPUs [ 0.061018] smpboot: Max logical packages: 1 [ 0.062013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144942] node 0 deferred pages initialised in 80ms [ 0.148152] devtmpfs: initialized [ 0.149208] x86/mm: Memory block size: 128MB [ 0.151386] gcov: version magic: 0x41383552 [ 0.154268] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156294] pinctrl core: initialized pinctrl subsystem [ 0.157170] [ 0.157848] ************************************************************* [ 0.158012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159012] ** ** [ 0.160015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161013] ** ** [ 0.162014] ** This means that this kernel is built to expose internal ** [ 0.163013] ** IOMMU data structures, which may compromise security on ** [ 0.164010] ** your system. ** [ 0.165010] ** ** [ 0.166011] ** If you see this message and you are not debugging the ** [ 0.167013] ** kernel, report this immediately to your vendor! ** [ 0.168013] ** ** [ 0.169012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170011] ************************************************************* [ 0.171637] NET: Registered protocol family 16 [ 0.173137] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182118] cpuidle: using governor menu [ 0.185176] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188493] PCI: Using configuration type 1 for base access [ 0.192135] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.201157] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.204025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.209034] cryptd: max_cpu_qlen set to 1000 [ 0.211206] ACPI: Added _OSI(Module Device) [ 0.215034] ACPI: Added _OSI(Processor Device) [ 0.216015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223501] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.229519] ACPI: Interpreter enabled [ 0.231063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233017] ACPI: Using IOAPIC for interrupt routing [ 0.235142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.239395] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.249280] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.251041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.254021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.258061] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.263413] acpiphp: Slot [2] registered [ 0.265153] acpiphp: Slot [5] registered [ 0.267126] acpiphp: Slot [6] registered [ 0.269114] acpiphp: Slot [3] registered [ 0.270100] acpiphp: Slot [4] registered [ 0.272102] acpiphp: Slot [7] registered [ 0.274095] acpiphp: Slot [8] registered [ 0.275127] acpiphp: Slot [9] registered [ 0.277085] acpiphp: Slot [10] registered [ 0.279110] acpiphp: Slot [11] registered [ 0.281103] acpiphp: Slot [12] registered [ 0.283087] acpiphp: Slot [13] registered [ 0.285109] acpiphp: Slot [14] registered [ 0.286116] acpiphp: Slot [15] registered [ 0.288098] acpiphp: Slot [16] registered [ 0.289099] acpiphp: Slot [17] registered [ 0.291109] acpiphp: Slot [18] registered [ 0.293108] acpiphp: Slot [19] registered [ 0.295089] acpiphp: Slot [20] registered [ 0.296078] acpiphp: Slot [21] registered [ 0.298135] acpiphp: Slot [22] registered [ 0.299142] acpiphp: Slot [23] registered [ 0.300087] acpiphp: Slot [24] registered [ 0.302102] acpiphp: Slot [25] registered [ 0.303071] acpiphp: Slot [26] registered [ 0.303981] acpiphp: Slot [27] registered [ 0.305084] acpiphp: Slot [28] registered [ 0.306079] acpiphp: Slot [29] registered [ 0.308106] acpiphp: Slot [30] registered [ 0.309088] acpiphp: Slot [31] registered [ 0.311072] PCI host bridge to bus 0000:00 [ 0.312017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326879] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330277] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.337534] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.341013] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.344014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.350451] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.352695] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.354030] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.357803] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.371484] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.375011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.380000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.386013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.394013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.408000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.416793] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.432014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.447016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.456000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.460406] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.463336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.465405] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.468227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.473082] iommu: Default domain type: Passthrough [ 0.474397] SCSI subsystem initialized [ 0.476094] ACPI: bus type USB registered [ 0.477090] usbcore: registered new interface driver usbfs [ 0.478064] usbcore: registered new interface driver hub [ 0.480079] usbcore: registered new device driver usb [ 0.481107] pps_core: LinuxPPS API ver. 1 registered [ 0.482006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.484042] PTP clock support registered [ 0.485211] EDAC MC: Ver: 3.0.0 [ 0.487389] PCI: Using ACPI for IRQ routing [ 0.489781] NetLabel: Initializing [ 0.491010] NetLabel: domain hash size = 128 [ 0.493010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.495079] NetLabel: unlabeled traffic allowed by default [ 0.498036] vgaarb: loaded [ 0.499293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.501013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.507862] clocksource: Switched to clocksource kvm-clock [ 0.612526] VFS: Disk quotas dquot_6.6.0 [ 0.614236] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.616690] *** VALIDATE ramfs *** [ 0.618018] *** VALIDATE hugetlbfs *** [ 0.620385] pnp: PnP ACPI init [ 0.622723] pnp: PnP ACPI: found 6 devices [ 0.643817] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.647315] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.649773] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.651909] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.653818] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.655593] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.658107] NET: Registered protocol family 2 [ 0.660562] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.665374] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.668419] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.672718] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.675834] TCP: Hash tables configured (established 65536 bind 65536) [ 0.678529] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.681210] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.684099] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.687353] NET: Registered protocol family 1 [ 0.689578] RPC: Registered named UNIX socket transport module. [ 0.691532] RPC: Registered udp transport module. [ 0.693920] RPC: Registered tcp transport module. [ 0.695691] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.698136] NET: Registered protocol family 44 [ 0.699798] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.702065] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.704778] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.707188] PCI: CLS 0 bytes, default 64 [ 0.709418] Unpacking initramfs... [ 2.093448] debug: unmapping init [mem 0xffff8caffcc64000-0xffff8caffffcffff] [ 2.098741] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.101843] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.105786] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.590080] Initialise system trusted keyrings [ 2.592051] Key type blacklist registered [ 2.594061] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.602757] zbud: loaded [ 2.604870] *** VALIDATE nfs *** [ 2.606137] *** VALIDATE nfs4 *** [ 2.607640] pstore: using deflate compression [ 2.611618] Platform Keyring initialized [ 2.718476] NET: Registered protocol family 38 [ 2.720504] Key type asymmetric registered [ 2.722345] Asymmetric key parser 'x509' registered [ 2.724616] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.728251] io scheduler mq-deadline registered [ 2.730427] io scheduler kyber registered [ 2.732442] io scheduler bfq registered [ 2.734636] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.737786] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.740227] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.742972] ACPI: Power Button [PWRF] [ 2.748042] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.754912] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.769346] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.797923] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.827027] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.833526] Non-volatile memory driver v1.3 [ 2.835503] Linux agpgart interface v0.103 [ 2.869364] virtio_blk virtio1: [vda] 147160 512-byte logical blocks (75.3 MB/71.9 MiB) [ 2.872339] vda: detected capacity change from 0 to 75345920 [ 2.889435] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.892061] vdb: detected capacity change from 0 to 1073741824 [ 2.899131] libphy: Fixed MDIO Bus: probed [ 2.908979] usbcore: registered new interface driver usbserial_generic [ 2.912247] usbserial: USB Serial support registered for generic [ 2.914548] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.917543] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.919071] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.921776] mousedev: PS/2 mouse device common for all mice [ 2.924306] rtc_cmos 00:05: RTC can wake from S4 [ 2.927577] rtc_cmos 00:05: registered as rtc0 [ 2.929403] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.929508] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.931652] intel_pstate: CPU model not supported [ 2.938113] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.940535] hid: raw HID events driver (C) Jiri Kosina [ 2.944017] usbcore: registered new interface driver usbhid [ 2.946082] usbhid: USB HID core driver [ 2.947672] drop_monitor: Initializing network drop monitor service [ 2.949679] Initializing XFRM netlink socket [ 2.949795] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.952091] NET: Registered protocol family 10 [ 2.956617] Segment Routing with IPv6 [ 2.958374] NET: Registered protocol family 17 [ 2.960794] mpls_gso: MPLS GSO support [ 2.965780] RAS: Correctable Errors collector initialized. [ 2.967345] AVX version of gcm_enc/dec engaged. [ 2.968646] AES CTR mode by8 optimization enabled [ 3.046869] sched_clock: Marking stable (3046788828, 0)->(3918053734, -871264906) [ 3.051238] registered taskstats version 1 [ 3.053750] Loading compiled-in X.509 certificates [ 3.055990] zswap: loaded using pool lzo/zbud [ 3.080986] Key type big_key registered [ 3.093609] Key type encrypted registered [ 3.095132] ima: No TPM chip found, activating TPM-bypass! [ 3.097140] ima: Allocated hash algorithm: sha1 [ 3.098259] ima: No architecture policies found [ 3.099723] evm: Initialising EVM extended attributes: [ 3.101294] evm: security.selinux [ 3.102519] evm: security.ima [ 3.103549] evm: security.capability [ 3.104698] evm: HMAC attrs: 0x1 [ 3.106804] rtc_cmos 00:05: setting system clock to 2026-09-11 12:47:41 UTC (1789130861) [ 3.112926] debug: unmapping init [mem 0xffffffff95003000-0xffffffff951fffff] [ 3.115252] debug: unmapping init [mem 0xffffffff93d82000-0xffffffff94058fff] [ 3.124101] Write protecting the kernel read-only data: 28672k [ 3.126985] debug: unmapping init [mem 0xffffffff92403000-0xffffffff925fffff] [ 3.129688] debug: unmapping init [mem 0xffffffff92d14000-0xffffffff92dfffff] [ 3.164089] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.170668] systemd[1]: Detected virtualization kvm. [ 3.172348] systemd[1]: Detected architecture x86-64. [ 3.174507] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.201103] systemd[1]: No hostname configured. [ 3.203521] systemd[1]: Set hostname to . [ 3.206707] random: systemd: uninitialized urandom read (16 bytes read) [ 3.210368] systemd[1]: Initializing machine ID from random generator. [ 3.252531] random: ln: uninitialized urandom read (6 bytes read) [ 3.334335] random: systemd: uninitialized urandom read (16 bytes read) [ 3.337530] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.342671] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.346187] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. [ 3.860220] random: fast init done Starting dracut pre-udev hook... [ 3.906849] device-mapper: uevent: version 1.0.3 [ 3.909078] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.595834] virtio_net virtio0 ens2: renamed from eth0 [ 4.691659] scsi host0: ata_piix [ 4.708185] scsi host1: ata_piix [ 4.709930] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.713069] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.973054] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.619246] random: crng init done [ 9.621566] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.911207] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.098346] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.360602] SELinux: Disabled at runtime. [ 11.421086] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.430648] systemd[1]: Detected virtualization kvm. [ 11.432727] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.900787] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.905392] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.911379] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.916764] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.919883] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.926597] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.943354] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ 12.039525] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.368923] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.632135] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.648346] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.734474] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.746648] EDAC sbridge: Ver: 1.1.2 [ 13.710049] Key type dns_resolver registered [ 14.012389] NFS: Registering the id_resolver key type [ 14.014378] Key type id_resolver registered [ 14.015743] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg332-client login: [ 75.025845] libcfs: loading out-of-tree module taints kernel. [ 75.203507] Key type ._llcrypt registered [ 75.229546] Key type .llcrypt registered [ 75.517648] hrtimer: interrupt took 4639044 ns [ 75.820899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 75.849385] alg: No test for adler32 (adler32-zlib) [ 77.399195] Lustre: Lustre: Build Version: 2.17.57_132_gd76390e [ 78.287727] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 80.063374] Key type lgssc registered [ 82.025634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 203.671038] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e [ 208.414455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.029307] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing check_logdir /tmp/testlogs/ [ 226.322212] Lustre: DEBUG MARKER: oleg332-client.virtnet: executing yml_node [ 229.343193] Lustre: lustre-OST0000-osc-ffff8cb045c27000: disconnect after 23s idle [ 230.949955] Lustre: DEBUG MARKER: Client: 2.17.57.132 [ 233.639973] Lustre: DEBUG MARKER: MDS: 2.17.57.132 [ 236.104825] Lustre: DEBUG MARKER: OSS: 2.17.57.132 [ 237.669437] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Sep 11 08:51:34 EDT 2026 [ 253.445109] Lustre: Unmounted lustre-client [ 288.267474] Key type lgssc unregistered [ 288.515948] LNet: 8419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.529627] LNetError: 8419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.549627] LNet: Removed LNI 192.168.203.32@tcp [ 289.343175] Key type .llcrypt unregistered [ 289.345446] Key type ._llcrypt unregistered [ 311.564564] Key type ._llcrypt registered [ 311.571417] Key type .llcrypt registered [ 311.824437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 311.858337] alg: No test for adler32 (adler32-zlib) [ 312.860719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 312.972443] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 313.019641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 317.432968] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 317.850339] LNet: 9307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.863950] LNetError: 9307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 317.893335] LNet: Removed LNI 192.168.203.32@tcp [ 319.010456] Key type .llcrypt unregistered [ 319.018265] Key type ._llcrypt unregistered [ 326.810100] Key type ._llcrypt registered [ 326.814907] Key type .llcrypt registered [ 328.062049] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 08:53:04 (1789131184) [ 328.446492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 328.461382] alg: No test for adler32 (adler32-zlib) [ 329.392061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.517794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.550802] LNet: 9722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.574646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.633445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 329.655949] LNet: 9729:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 336.322817] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 08:53:13 (1789131193) [ 336.497912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.532762] LNet: 10305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.563782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.644052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 342.931720] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 08:53:19 (1789131199) [ 343.053855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.088157] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.130305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.180410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 349.312366] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 08:53:26 (1789131206) [ 349.432835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.462666] LNet: 11473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 349.484484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.604453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 355.969978] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 08:53:33 (1789131213) [ 356.232780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.272403] LNet: 12064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.296380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.401864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 356.793855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 362.277578] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 08:53:39 (1789131219) [ 362.466666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.519598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.632241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.098071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.510973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 369.970813] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 08:53:47 (1789131227) [ 370.063642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.088367] LNet: 13245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.095674] LNet: 13245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 370.120349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.224947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 376.373355] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 08:53:53 (1789131233) [ 376.455904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.501219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.580210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 376.608145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.025285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 377.052255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.417606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 377.456719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 377.791641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 377.833679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 378.350955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 378.399461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.834392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 378.885910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.255346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 379.280609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.584938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 379.610995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 379.928583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 379.961579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 380.367669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 380.408778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 380.810620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 380.837802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 381.189712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 381.224974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 381.572613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 381.608791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 387.736749] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 08:54:04 (1789131244) [ 387.976497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.015303] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.030161] LNet: 14517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 388.076722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.187131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 388.233466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 394.788410] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 08:54:12 (1789131252) [ 394.883292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.919862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.989841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 395.030177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.931732] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 08:54:18 (1789131258) [ 401.134956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.180996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.260782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 401.298720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 408.427712] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 08:54:25 (1789131265) [ 408.530944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.587310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.678257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 408.715602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.670516] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 08:54:31 (1789131271) [ 414.952231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.998256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.091529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 415.148850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 420.826667] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 08:54:37 (1789131277) [ 420.909106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.924346] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 420.932591] LNet: 17442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 420.962212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.045743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 421.087691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 427.363434] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 08:54:44 (1789131284) [ 427.480672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.543513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.653879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.161508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 434.036897] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 08:54:51 (1789131291) [ 434.211245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.271576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.495738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.822248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.684602] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 08:54:58 (1789131298) [ 440.895967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.930329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.060496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 441.217706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.359155] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 08:55:04 (1789131304) [ 447.542608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.583661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.054388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.372311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.117989] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 08:55:12 (1789131312) [ 455.257929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.312449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.411252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.498899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.903601] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 08:55:18 (1789131318) [ 461.076096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.153476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.228482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 461.272538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 466.972365] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 08:55:24 (1789131324) [ 467.120211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.159183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.298577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 472.906846] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 08:55:30 (1789131330) [ 473.066118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.105108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.182279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 478.590902] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 08:55:35 (1789131335) [ 478.682837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.742059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.845937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 479.205273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 484.918215] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 08:55:42 (1789131342) [ 485.004233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.045202] LNet: 23470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 485.050213] LNet: 23470:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 485.072225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.136993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 485.166140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 490.847373] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 08:55:48 (1789131348) [ 490.925551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.979838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.083707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 491.115423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 496.918721] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 08:55:54 (1789131354) [ 497.081184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.120150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.163785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 497.189783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 502.680992] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 08:55:59 (1789131359) [ 502.865956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.920934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.987565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 503.027701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 508.843481] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 08:56:05 (1789131365) [ 508.911207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.970723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.012932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 509.078632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 509.100771] LNetError: 25832:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 509.158503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 509.206342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 514.614304] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 08:56:11 (1789131371) [ 514.697328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 514.753679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.847626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UOi5bS4xuF [ 514.934309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UOi5bS4xuF [ 514.986936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.UOi5bS4xuF [ 519.973111] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 08:56:17 (1789131377) [ 520.033436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.096464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.175238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 520.202925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 520.252405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 525.233160] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 08:56:22 (1789131382) [ 525.470317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.502706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.589382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 525.617887] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 526.840295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 527.909663] Lustre: Lustre: Build Version: 2.17.57_132_gd76390e [ 529.728100] Key type lgssc registered [ 530.758716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 534.690971] LNetError: 27593:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.32@tcp99 has no available nets [ 537.131773] Key type lgssc unregistered [ 537.459203] LNetError: 28703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.472723] LNet: Removed LNI 192.168.203.32@tcp [ 537.996730] Key type .llcrypt unregistered [ 538.003481] Key type ._llcrypt unregistered [ 543.326023] Key type ._llcrypt registered [ 543.327487] Key type .llcrypt registered [ 544.172404] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 08:56:41 (1789131401) [ 544.477943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 544.486432] alg: No test for adler32 (adler32-zlib) [ 545.382431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.428494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 547.776205] LNet: Message forwarding will be enabled in 2s [ 550.879236] LNet: Message forwarding enabled [ 551.193188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.214193] LNet: Message forwarding will stop in 2s to 6s [ 555.813185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 556.978047] LNet: Message forwarding disabled [ 559.518728] LNet: Message forwarding will be enabled in 2s [ 563.106112] LNet: Message forwarding enabled [ 566.920688] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 08:57:04 (1789131424) [ 567.160681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.204392] LNet: 30111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 568.735640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.793470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 574.449724] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 08:57:11 (1789131431) [ 574.566954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.587810] LNet: 30686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.613794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.669928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 574.745995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 574.775290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 574.802667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 574.826918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 574.860838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 574.890785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 574.922221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 574.968875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 575.017053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 575.055976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 575.106885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 575.158629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 575.200674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 575.250868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 575.283751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 575.309376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 575.351627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 575.380589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 575.405669] LNetError: 30735:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 581.434831] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 08:57:18 (1789131438) [ 581.512950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.537007] LNet: 31299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.572583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.668187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 587.171685] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 08:57:24 (1789131444) [ 588.877843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg332-server [ 590.321371] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 08:57:27 (1789131447) [ 590.560465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.578453] LNet: 32237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 590.598352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.650518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.673602] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 591.377990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[33-37]@tcp [ 591.397963] LNet: 32370:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 591.441245] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 591.454669] LNetError: 32241:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.34@tcp has gone from up to down [ 592.446563] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.33@tcp is being used as a gateway but routing feature is not turned on [ 592.473498] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 593.454487] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.37@tcp is being used as a gateway but routing feature is not turned on [ 593.469470] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 595.490761] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.36@tcp is being used as a gateway but routing feature is not turned on [ 595.516316] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 8 previous similar messages [ 597.275805] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 08:57:34 (1789131454) [ 598.961789] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg332-server [ 599.593131] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 599.614649] LNetError: 32241:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 600.479603] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 08:57:37 (1789131457) [ 600.592188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.627637] LNetError: 32252:0:(lib-move.c:4801:lnet_parse()) 192.168.203.34@tcp, src 192.168.203.34@tcp: Dropping REPLY (error -108 looking up sender) [ 600.634246] LNet: 33302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 600.644123] LNetError: 32252:0:(lib-move.c:4801:lnet_parse()) Skipped 1 previous similar message [ 600.681272] LNetError: 33302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.705416] LNet: Removed LNI 192.168.203.32@tcp [ 601.721318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.766870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 601.788080] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 602.370855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[33-38/2]@tcp [ 602.395141] LNet: 33424:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 602.403310] LNet: 33424:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 602.441628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[33-38/2]@tcp [ 602.476090] LNetError: 33311:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.37@tcp has gone from up to down [ 602.490047] LNetError: 33311:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 608.386460] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 08:57:45 (1789131465) [ 608.481330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.529918] LNetError: 34012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.560148] LNet: Removed LNI 192.168.203.32@tcp [ 608.606683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.709758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 608.760776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.780958] LNet: 34030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.789142] LNet: 34030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 608.816425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 608.880396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 608.941701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 608.973290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.043054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 609.107140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.173446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.239811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.292579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.327296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.374547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 609.452961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.507816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.625412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 609.712949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.763920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.827262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 609.937240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.978923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.073619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.161880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.196429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.256778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 610.361561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.430828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.542918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.672588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.704982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.755888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 610.814403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.849099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.925730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 616.166646] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 08:57:53 (1789131473) [ 617.957275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg332-server [ 619.186612] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 08:57:56 (1789131476) [ 620.620910] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg332-server [ 621.903701] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 08:57:59 (1789131479) [ 622.553905] Key type .llcrypt unregistered [ 622.556123] Key type ._llcrypt unregistered [ 627.602847] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 627.679780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 627.688839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 627.692794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 630.252499] Key type ._llcrypt registered [ 630.257965] Key type .llcrypt registered [ 630.472255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 630.488448] alg: No test for adler32 (adler32-zlib) [ 631.361144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.408425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 631.434930] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 631.933531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 631.952632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 633.712873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 633.731664] LNet: Removed LNI 10.1.2.3@tcp [ 633.827749] LNet: 36144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.837516] LNetError: 36144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 633.851511] LNet: Removed LNI 192.168.203.32@tcp [ 634.396976] Key type .llcrypt unregistered [ 634.398761] Key type ._llcrypt unregistered [ 643.591629] Key type ._llcrypt registered [ 643.593915] Key type .llcrypt registered [ 644.764787] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 08:58:21 (1789131501) [ 645.200546] Key type .llcrypt unregistered [ 645.204196] Key type ._llcrypt unregistered [ 650.090521] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 650.238281] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 650.242393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 650.249618] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 652.779068] Key type ._llcrypt registered [ 652.780624] Key type .llcrypt registered [ 652.915693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 652.925943] alg: No test for adler32 (adler32-zlib) [ 653.795959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.857665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.900099] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 654.456678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 654.487691] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 656.585770] LNet: 37639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.592342] LNetError: 37639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 656.599875] LNet: Removed LNI 10.1.2.3@tcp [ 656.618934] LNet: Removed LNI 192.168.203.32@tcp [ 657.165270] Key type .llcrypt unregistered [ 657.167719] Key type ._llcrypt unregistered [ 665.831576] Key type ._llcrypt registered [ 665.834583] Key type .llcrypt registered [ 666.989503] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 08:58:44 (1789131524) [ 667.501955] Key type .llcrypt unregistered [ 667.505079] Key type ._llcrypt unregistered [ 672.456096] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 672.585618] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 672.595130] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 672.599576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 675.137625] Key type ._llcrypt registered [ 675.140366] Key type .llcrypt registered [ 675.328664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.342162] alg: No test for adler32 (adler32-zlib) [ 676.253906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.290572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 676.314851] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 677.878981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 677.900102] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.807762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 679.835937] LNet: Removed LNI 10.1.2.3@tcp [ 679.845767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 679.869678] LNet: Removed LNI 10.1.2.31@tcp [ 679.886929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 679.908221] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 680.603710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 680.622923] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 681.528853] LNet: 39569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.542985] LNetError: 39569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 681.553499] LNet: Removed LNI 10.1.2.31@tcp [ 681.565898] LNet: Removed LNI 10.1.2.3@tcp [ 682.171056] Key type .llcrypt unregistered [ 682.173868] Key type ._llcrypt unregistered [ 692.725835] Key type ._llcrypt registered [ 692.728695] Key type .llcrypt registered [ 693.875278] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 08:59:10 (1789131550) [ 694.239386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 694.258492] alg: No test for adler32 (adler32-zlib) [ 695.159896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.260598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 695.285628] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 696.051674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 696.079513] LNet: 40526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.084932] LNetError: 40526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 696.105864] LNet: Removed LNI 192.168.203.32@tcp [ 696.134115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.184395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.32@tcp [ 696.217494] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 697.250379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.32@tcp [ 697.322443] LNet: Removed LNI 192.168.203.32@tcp [ 697.417368] LNet: 40661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.367386] Key type .llcrypt unregistered [ 698.372828] Key type ._llcrypt unregistered [ 710.352449] Key type ._llcrypt registered [ 710.359422] Key type .llcrypt registered [ 711.697899] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 08:59:28 (1789131568) [ 713.735678] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg332-server [ 715.396977] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 08:59:32 (1789131572) [ 715.984454] Key type .llcrypt unregistered [ 715.987215] Key type ._llcrypt unregistered [ 721.925496] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 722.102932] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.106974] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.110194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.513743] Key type ._llcrypt registered [ 724.518302] Key type .llcrypt registered [ 724.726148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.750878] alg: No test for adler32 (adler32-zlib) [ 725.688282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.746068] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 726.401213] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 728.371276] LNet: 42506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.376505] LNetError: 42506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.382870] LNet: Removed LNI 192.168.203.32@tcp1 [ 728.390878] LNet: Removed LNI 10.1.2.3@tcp2 [ 728.940710] Key type .llcrypt unregistered [ 728.946845] Key type ._llcrypt unregistered [ 739.453907] Key type ._llcrypt registered [ 739.458642] Key type .llcrypt registered [ 740.748868] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 08:59:57 (1789131597) [ 741.182242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.202683] alg: No test for adler32 (adler32-zlib) [ 742.108934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.161947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.179652] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 743.017812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.053622] LNet: 43480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.063847] LNetError: 43480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.080128] LNet: Removed LNI 192.168.203.32@tcp [ 743.118418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.196551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 743.237933] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 748.910331] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:00:06 (1789131606) [ 749.146583] LNet: 44071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.155880] LNetError: 44071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.181945] LNet: Removed LNI 192.168.203.32@tcp [ 749.801644] Key type .llcrypt unregistered [ 749.805027] Key type ._llcrypt unregistered [ 754.451503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.602993] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.611088] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.616491] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 757.080251] Key type ._llcrypt registered [ 757.083384] Key type .llcrypt registered [ 757.222172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 757.232871] alg: No test for adler32 (adler32-zlib) [ 758.086921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.166599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 758.209919] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 758.793593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.831966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.902140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.929167] LNet: 44817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.935618] LNetError: 44817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 760.939738] LNet: Removed LNI 10.1.2.3@tcp [ 760.952442] LNet: Removed LNI 192.168.203.32@tcp [ 760.966804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.025472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.054678] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 761.061503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.618954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.639760] LNet: 44845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.648965] LNetError: 44845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 761.654158] LNetError: 44845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.658539] LNet: Removed LNI 10.1.2.3@tcp [ 761.670903] LNet: Removed LNI 192.168.203.32@tcp [ 761.685362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.740560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.763359] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 761.767696] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 762.058874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 762.081197] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 762.907602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 762.919091] LNet: 45007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.922725] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 762.926153] LNetError: 45007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 762.929373] LNet: Removed LNI 10.1.2.3@tcp [ 762.930644] LNet: Removed LNI 192.168.203.32@tcp [ 762.940074] LNet: Removed LNI 192.168.203.32@tcp2 [ 762.961237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.993867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 763.014058] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 763.018625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 763.022516] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 768.530037] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:00:25 (1789131625) [ 768.682888] LNet: 45604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.687832] LNetError: 45604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 768.692593] LNetError: 45604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 768.698755] LNet: Removed LNI 10.1.2.3@tcp [ 768.700921] LNet: Removed LNI 192.168.203.32@tcp [ 768.710322] LNet: Removed LNI 192.168.203.32@tcp2 [ 769.351450] Key type .llcrypt unregistered [ 769.356438] Key type ._llcrypt unregistered [ 775.101093] Key type ._llcrypt registered [ 775.102841] Key type .llcrypt registered [ 775.531425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 775.580479] alg: No test for adler32 (adler32-zlib) [ 776.543161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.593384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 776.608875] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 777.280979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.306536] LNet: 46003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.318610] LNetError: 46003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.355263] LNet: Removed LNI 192.168.203.32@tcp [ 777.394150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.501494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 777.537630] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 778.043159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.066580] LNet: 46033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.075856] LNetError: 46033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.092441] LNet: Removed LNI 192.168.203.32@tcp [ 778.108843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.160500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.180189] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 778.484795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.524307] LNet: Removed LNI 192.168.203.32@tcp [ 778.552355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.658416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.674954] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 778.957788] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 779.075312] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 779.083323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 779.085691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.465656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.525878] LNet: 46140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.532757] LNet: 46140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 781.542018] LNetError: 46140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.551069] LNetError: 46140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 781.569105] LNet: Removed LNI 192.168.203.32@tcp [ 781.593532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.650473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 781.673483] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 787.835841] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:00:44 (1789131644) [ 787.965405] LNet: 46733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.974627] LNetError: 46733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.004548] LNet: Removed LNI 192.168.203.32@tcp [ 788.674864] Key type .llcrypt unregistered [ 788.676797] Key type ._llcrypt unregistered [ 794.185904] Key type ._llcrypt registered [ 794.187440] Key type .llcrypt registered [ 794.356156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 794.364570] alg: No test for adler32 (adler32-zlib) [ 795.224256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.256771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 795.286767] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 795.961206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.981441] LNet: 47140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.989237] LNetError: 47140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.002644] LNet: Removed LNI 192.168.203.32@tcp [ 796.022497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.136803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 796.163313] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 801.765532] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:00:58 (1789131658) [ 801.874724] LNet: 47732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.880583] LNetError: 47732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.892380] LNet: Removed LNI 192.168.203.32@tcp [ 802.317607] Key type .llcrypt unregistered [ 802.320080] Key type ._llcrypt unregistered [ 807.070119] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 807.223349] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 807.229058] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 807.232758] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 809.689607] Key type ._llcrypt registered [ 809.692382] Key type .llcrypt registered [ 809.891914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 809.918185] alg: No test for adler32 (adler32-zlib) [ 810.837504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.891433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.916857] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 811.637413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 811.669309] LNet: 48193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 811.676195] LNetError: 48193:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 811.701892] LNet: Removed LNI 192.168.203.32@tcp [ 811.719090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.785374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 811.803811] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 817.103532] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:01:14 (1789131674) [ 817.229424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.249612] LNet: 48791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.259046] LNetError: 48791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.281941] LNet: Removed LNI 192.168.203.32@tcp [ 817.306445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.419550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 822.416529] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:01:19 (1789131679) [ 822.686914] LNet: 49374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.197706] Key type .llcrypt unregistered [ 823.201103] Key type ._llcrypt unregistered [ 828.694285] Key type ._llcrypt registered [ 828.698207] Key type .llcrypt registered [ 828.947743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.971939] alg: No test for adler32 (adler32-zlib) [ 829.895152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.948284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.981903] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 830.743871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.764791] LNet: 49776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.777048] LNetError: 49776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.803476] LNet: Removed LNI 192.168.203.32@tcp [ 830.839658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.926180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 830.961311] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 837.254820] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:01:34 (1789131694) [ 837.478425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.500852] LNet: 50369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.508840] LNetError: 50369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.534433] LNet: Removed LNI 192.168.203.32@tcp [ 837.567472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.635995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 842.944700] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:01:40 (1789131700) [ 844.138866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 845.800555] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:01:42 (1789131702) [ 847.057780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 848.381281] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:01:45 (1789131705) [ 850.082044] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 851.734257] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:01:48 (1789131708) [ 853.063861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 854.506478] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:01:51 (1789131711) [ 856.010213] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 857.472519] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:01:54 (1789131714) [ 857.558645] LNet: 52411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.272374] Key type .llcrypt unregistered [ 858.274156] Key type ._llcrypt unregistered [ 863.651861] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 863.911751] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 863.932946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 863.935760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.508463] Key type ._llcrypt registered [ 866.510277] Key type .llcrypt registered [ 866.712942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.732289] alg: No test for adler32 (adler32-zlib) [ 867.628383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.702476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 867.719824] LNet: Added LNI 192.168.203.32@tcp [8/512/0/180] [ 867.724833] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 867.726236] LNet: 52742:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 867.827559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.843976] LNet: 52761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.849931] LNetError: 52761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.856115] LNet: Removed LNI 10.1.2.3@tcp [ 867.859995] LNetError: 52761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.881597] LNet: Removed LNI 192.168.203.32@tcp [ 867.939866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.031134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.062571] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 868.130438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.158439] LNet: 52785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.163353] LNetError: 52785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.180258] LNet: Removed LNI 192.168.203.32@tcp [ 868.196140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.270984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.295382] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 868.305810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.309802] LNet: 52793:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 868.401722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.448262] LNet: 52811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.454825] LNetError: 52811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.462551] LNet: Removed LNI 10.1.2.3@tcp [ 868.465686] LNetError: 52811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.480713] LNet: Removed LNI 192.168.203.32@tcp [ 868.498279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.565182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.583897] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 868.590797] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 868.593912] LNet: 52819:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 868.756876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 868.775156] LNet: 52845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.780514] LNetError: 52845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 868.785745] LNet: Removed LNI 10.1.2.3@tcp [ 868.788820] LNetError: 52845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.808672] LNet: Removed LNI 192.168.203.32@tcp [ 868.836324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.913794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 868.960891] LNet: Added LNI 192.168.203.32@tcp [8/256/0/182] [ 868.968546] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 869.080547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.111606] LNet: 52873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.118341] LNetError: 52873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 869.123465] LNet: Removed LNI 10.1.2.3@tcp [ 869.126196] LNetError: 52873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.137710] LNet: Removed LNI 192.168.203.32@tcp [ 869.153335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.210746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 869.229831] LNet: Added LNI 192.168.203.32@tcp [8/256/0/182] [ 869.234609] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 869.236652] LNet: 52881:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 869.291566] LNet: Removed LNI 10.1.2.3@tcp [ 869.305940] LNet: Removed LNI 192.168.203.32@tcp [ 869.966897] Key type .llcrypt unregistered [ 869.979368] Key type ._llcrypt unregistered [ 880.729481] Key type ._llcrypt registered [ 880.732969] Key type .llcrypt registered [ 881.980349] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:02:18 (1789131738) [ 883.265496] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 884.779632] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:02:21 (1789131741) [ 885.170276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 885.190608] alg: No test for adler32 (adler32-zlib) [ 886.115621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.198366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.245514] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 887.083520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.32 [ 887.107713] LNet: 54149:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 887.115763] LNet: Removed LNI 192.168.203.32@tcp [ 892.881781] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:02:30 (1789131750) [ 893.040203] LNet: 54717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.046685] LNetError: 54717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.071716] LNet: Removed LNI 192.168.203.32@tcp [ 893.760562] Key type .llcrypt unregistered [ 893.770324] Key type ._llcrypt unregistered [ 899.934090] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 900.113878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 902.657742] Key type ._llcrypt registered [ 902.661897] Key type .llcrypt registered [ 902.831232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.851955] alg: No test for adler32 (adler32-zlib) [ 904.142629] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 909.666647] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:02:47 (1789131767) [ 909.787423] LNet: 55634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.797441] LNetError: 55634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 909.837241] LNet: Removed LNI 10.1.2.3@tcp [ 910.437917] Key type .llcrypt unregistered [ 910.443514] Key type ._llcrypt unregistered [ 915.808794] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.013379] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 916.024221] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 916.045767] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.668308] Key type ._llcrypt registered [ 918.670849] Key type .llcrypt registered [ 918.883942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.907224] alg: No test for adler32 (adler32-zlib) [ 919.853676] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 925.260302] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:03:02 (1789131782) [ 926.599891] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 928.029632] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:03:05 (1789131785) [ 928.103092] LNet: 56819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.111406] LNetError: 56819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 928.125074] LNet: Removed LNI 10.1.2.3@tcp [ 928.624297] Key type .llcrypt unregistered [ 928.626165] Key type ._llcrypt unregistered [ 934.799248] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 935.066935] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 935.074608] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 935.079859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 936.775286] Key type ._llcrypt registered [ 936.778436] Key type .llcrypt registered [ 937.030441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.056409] alg: No test for adler32 (adler32-zlib) [ 937.949448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.021726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 944.848851] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:03:22 (1789131802) [ 945.383723] LNet: 58051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.388970] LNetError: 58051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 945.402042] LNet: Removed LNI 10.1.2.3@tcp [ 945.904198] Key type .llcrypt unregistered [ 945.906741] Key type ._llcrypt unregistered [ 951.532206] Key type ._llcrypt registered [ 951.533931] Key type .llcrypt registered [ 951.745694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.766337] alg: No test for adler32 (adler32-zlib) [ 952.741719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 952.793729] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 960.414071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 961.052256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 961.088830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 961.122956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 961.138764] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 961.382297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 961.414490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 961.472234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.511619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.561193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.635561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 961.674967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 961.719263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 961.745668] LNet: 58492:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 961.770520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 961.851896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 961.913659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 961.951609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 961.984366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 962.017331] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 962.023147] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 962.027277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 962.074171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 962.590281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 962.635419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 962.682216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 962.954723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 962.975974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 963.005825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 963.035449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 963.083390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 963.138169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 963.188119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 963.238406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 963.265309] LNet: 58602:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 963.292280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 963.327209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 963.373345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.411399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.452048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.525237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.560567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 964.152114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 964.196901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 964.241368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 964.277087] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 964.288167] LNet: Skipped 1 previous similar message [ 964.492040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 964.523523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 964.550566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 964.577227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 964.600202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 964.626591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 964.647827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 964.673905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 964.690962] LNet: 58711:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 964.707159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 964.740363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 964.767566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 964.794727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 964.833858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 964.872251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 964.930290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 965.505258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 965.559119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 965.606374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 965.890851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 965.919214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 965.946428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 965.969810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 965.994305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 966.023950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 966.055665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 966.105923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 966.136938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 966.170992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 966.211195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 966.250279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 966.292365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 966.315808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 966.344769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 966.883882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 966.920557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 966.974879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 966.997596] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 967.005249] LNet: Skipped 1 previous similar message [ 967.208853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 967.256813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 967.282916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 967.310920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 967.348255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 967.382719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 967.418132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 967.467323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 967.485902] LNet: 58929:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.491728] LNet: 58929:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 967.508744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 967.556872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 967.586926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 967.624355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 967.655214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 967.679651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 967.701927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 968.375589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 968.424609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 968.463836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 968.713889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 968.737547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 968.765544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 968.799543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 968.835869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 968.867752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 968.900699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 968.938974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 968.975675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 969.015993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 969.055015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 969.101189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.149202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 969.183193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 969.225837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 969.546134] LNet: 59083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.567248] LNetError: 59083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.595431] LNet: Removed LNI 192.168.203.32@tcp [ 969.963328] Key type .llcrypt unregistered [ 969.966519] Key type ._llcrypt unregistered [ 985.512151] Key type ._llcrypt registered [ 985.514908] Key type .llcrypt registered [ 986.478309] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:04:03 (1789131843) [ 987.108545] Key type .llcrypt unregistered [ 987.110547] Key type ._llcrypt unregistered [ 991.843411] Key type ._llcrypt registered [ 991.845270] Key type .llcrypt registered [ 992.138728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 992.157601] alg: No test for adler32 (adler32-zlib) [ 993.038607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 993.059261] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1000.987485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1010.630699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.679333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1010.702556] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1011.629584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 1011.670253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 1 [ 1011.716982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 1011.758765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 1 [ 1011.808749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_interrupt -r 1 [ 1011.880783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 1 [ 1011.922107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_interrupt -r 1 [ 1011.960207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 1 [ 1012.008637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1012.031962] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1012.038444] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1012.369755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.457740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1012.543490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.582313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.615606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1012.647940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1012.682674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1012.720734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1012.754465] LNet: 60554:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.788722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1012.817736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1012.846840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1012.873210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1012.908844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.960565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1012.993316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1013.714528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 1013.752222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 1 [ 1013.803670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 1013.852319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 1 [ 1013.908791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_dropped -r 1 [ 1013.939811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 1 [ 1013.998289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_dropped -r 1 [ 1014.039960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 1 [ 1014.084222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1014.102974] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1014.116603] LNet: Skipped 2 previous similar messages [ 1014.381437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.422563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1014.470595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.545777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.578784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.624621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.666703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.703449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.732246] LNet: 60676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.753591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.806309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.848193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.894368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.931290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.971695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.025197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1015.458816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 1015.494510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 1 [ 1015.535648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 1015.577303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 1 [ 1015.628950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_aborted -r 1 [ 1015.659359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 1 [ 1015.702523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_aborted -r 1 [ 1015.740355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 1 [ 1015.784605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1015.807705] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1015.815790] LNet: Skipped 2 previous similar messages [ 1016.088362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.117966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.146847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.185360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.226714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.272847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.305829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.328757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.354985] LNet: 60797:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1016.383471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.430727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.495808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.536438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.597804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.642681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.697596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1017.188712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 1017.216714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 1 [ 1017.249525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 1017.286899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 1 [ 1017.334788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_no_route -r 1 [ 1017.381826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 1 [ 1017.419488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_no_route -r 1 [ 1017.465078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 1 [ 1017.498615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1017.712386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.759169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.806733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.855976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.890708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.916297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.942399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.966102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.992904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1018.011077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1018.036432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.068373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.104787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.135251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.175924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1018.767476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 1018.794244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 1 [ 1018.820325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 1018.848905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 1 [ 1018.887839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_timeout -r 1 [ 1018.952511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 1 [ 1019.005365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_timeout -r 1 [ 1019.080570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 1 [ 1019.120992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1019.136412] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1019.142187] LNet: Skipped 5 previous similar messages [ 1019.402856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.453514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.503600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.553918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.626119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.667605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.699174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.747730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.759429] LNet: 61039:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1019.764093] LNet: 61039:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1019.774745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.793831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.809813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.832340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.853257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.877651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.896730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1020.304868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 1020.337920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_error -r 1 [ 1020.370343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 1020.411124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_error -r 1 [ 1020.450197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_error -r 1 [ 1020.506915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_error -r 1 [ 1020.556652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_error -r 1 [ 1020.592801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_error -r 1 [ 1020.643895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1020.889654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1020.921622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1020.946931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1020.978822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.005251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.031765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.073939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.132887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.186440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1021.223913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1021.255767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1021.326856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.364664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.394996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.427722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1026.669818] LNet: 61241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.685126] LNetError: 61241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1026.705641] LNet: Removed LNI 192.168.203.32@tcp [ 1027.761398] LNet: Removed LNI 192.168.203.32@tcp1 [ 1028.154841] Key type .llcrypt unregistered [ 1028.156772] Key type ._llcrypt unregistered [ 1044.483905] Key type ._llcrypt registered [ 1044.485987] Key type .llcrypt registered [ 1045.363459] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:05:02 (1789131902) [ 1046.045577] Key type .llcrypt unregistered [ 1046.048735] Key type ._llcrypt unregistered [ 1052.290156] Key type ._llcrypt registered [ 1052.291929] Key type .llcrypt registered [ 1052.462849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.484706] alg: No test for adler32 (adler32-zlib) [ 1053.440827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1053.465583] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1060.031724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1060.660907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 1060.698674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 1060.729593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1060.746402] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1061.066674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.129501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1061.171715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.209525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.239546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.268182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.306372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1061.341816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1061.365469] LNet: 62495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1061.380827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1061.411705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1061.436620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1061.462536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1061.496685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1061.523843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1061.555594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1062.024711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 1062.066313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 1062.112434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1062.146351] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1062.375558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1062.402386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1062.437795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1062.469384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1062.517355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1062.556897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1062.606081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.669638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.709274] LNet: 62604:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.736448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.774647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.809334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.844584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.869315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.904375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.940368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1063.422379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 1063.454646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 1063.525378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1063.558865] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1063.821809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.864530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.900349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.931188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.978744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1064.031735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1064.080880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1064.116173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1064.133576] LNet: 62713:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1064.159197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1064.195866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1064.242360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1064.275322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1064.329123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.359731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.388721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1064.726347] LNet: 62758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.732042] LNetError: 62758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.749284] LNet: Removed LNI 192.168.203.32@tcp [ 1065.085927] Key type .llcrypt unregistered [ 1065.090292] Key type ._llcrypt unregistered [ 1079.387435] Key type ._llcrypt registered [ 1079.390574] Key type .llcrypt registered [ 1080.588434] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:05:37 (1789131937) [ 1081.407546] Key type .llcrypt unregistered [ 1081.411563] Key type ._llcrypt unregistered [ 1087.623529] Key type ._llcrypt registered [ 1087.627226] Key type .llcrypt registered [ 1087.804564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.824687] alg: No test for adler32 (adler32-zlib) [ 1088.702777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1088.719528] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1095.573897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1104.649701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.700372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1104.729868] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1106.007726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 1106.054540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 1 [ 1106.089608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 1106.121402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 1 [ 1106.152745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_dropped -r 1 [ 1106.183904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 1 [ 1106.213348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_dropped -r 1 [ 1106.251388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 1 [ 1106.283775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1106.298260] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1106.304328] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 1106.473436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.521331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.558548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.614433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.649106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.677593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.705387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.746505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.769214] LNet: 64229:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.786713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.826755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.855311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.886933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.917400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.944676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.986241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.023351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.496140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 1107.526663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_error -r 1 [ 1107.562452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 1107.601366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_error -r 1 [ 1107.638974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_error -r 1 [ 1107.681656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_error -r 1 [ 1107.725202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_error -r 1 [ 1107.766778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_error -r 1 [ 1107.818272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1107.844600] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1107.851839] LNet: Skipped 2 previous similar messages [ 1108.058793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.090282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.115358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.151792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.177688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.206805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.234671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.262836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.282867] LNet: 64352:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.301900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.342327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.373242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.405531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.442745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.505547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.567699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1108.595442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.126055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 1109.185727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_timeout -r 1 [ 1109.240346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 1109.279722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_timeout -r 1 [ 1109.323205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_timeout -r 1 [ 1109.368374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_timeout -r 1 [ 1109.414332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_timeout -r 1 [ 1109.453618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_timeout -r 1 [ 1109.485269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.132@tcp [ 1109.517221] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1109.523368] LNet: Skipped 1 previous similar message [ 1109.746368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.778904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.803495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.837449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.882728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.921696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.972260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.996595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.027328] LNet: 64475:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.049831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.081801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.114578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.147467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.173821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.226315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.274284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.320545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.328444] LNet: 64557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.337286] LNetError: 64557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.344227] LNet: Removed LNI 192.168.203.32@tcp [ 1114.364265] LNet: Removed LNI 192.168.203.32@tcp1 [ 1114.659481] Key type .llcrypt unregistered [ 1114.662532] Key type ._llcrypt unregistered [ 1128.531114] Key type ._llcrypt registered [ 1128.538748] Key type .llcrypt registered [ 1129.540475] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:06:26 (1789131986) [ 1130.747978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1132.584313] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:06:29 (1789131989) [ 1133.472473] Key type .llcrypt unregistered [ 1133.474546] Key type ._llcrypt unregistered [ 1139.406366] Key type ._llcrypt registered [ 1139.407992] Key type .llcrypt registered [ 1139.667426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.683076] alg: No test for adler32 (adler32-zlib) [ 1140.640342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1140.677199] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1148.515882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1149.017223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1149.053918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1149.096356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1149.120220] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1149.371499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1149.395242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1149.418888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1149.455555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1149.492543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1149.523213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1149.570525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1149.597502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1149.611336] LNet: 66104:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1149.627428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1149.652770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1149.688300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1149.713996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1149.739980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.770484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1150.111870] LNet: 66147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.118619] LNetError: 66147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.143638] LNet: Removed LNI 192.168.203.32@tcp [ 1150.553562] Key type .llcrypt unregistered [ 1150.556126] Key type ._llcrypt unregistered [ 1169.123456] Key type ._llcrypt registered [ 1169.125965] Key type .llcrypt registered [ 1169.423575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1169.464035] alg: No test for adler32 (adler32-zlib) [ 1170.352454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1170.392672] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1178.636359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1187.692894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1187.769830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1187.807753] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1189.139697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1189.170874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e network_timeout -r 1 [ 1189.204495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1189.236716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e network_timeout -r 1 [ 1189.285889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e network_timeout -r 1 [ 1189.328634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e network_timeout -r 1 [ 1189.356327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e network_timeout -r 1 [ 1189.387808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e network_timeout -r 1 [ 1189.417818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1189.441169] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1189.628795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.674857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.725513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.776546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.844578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.857902] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1189.866264] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 1189.891990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.933902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.981924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.008945] LNet: 67030:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.034159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.071887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.105213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.140950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.186949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.222178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.971493] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1195.474711] LNet: 67107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.484410] LNetError: 67107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.499145] LNet: Removed LNI 192.168.203.32@tcp [ 1195.516552] LNet: Removed LNI 192.168.203.32@tcp1 [ 1195.964943] Key type .llcrypt unregistered [ 1195.968589] Key type ._llcrypt unregistered [ 1212.070681] Key type ._llcrypt registered [ 1212.079160] Key type .llcrypt registered [ 1213.943127] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:07:50 (1789132070) [ 1214.195337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1214.203598] alg: No test for adler32 (adler32-zlib) [ 1215.150483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.197850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1215.219715] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1216.155243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1216.191964] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1217.071170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1217.160953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1217.208898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1217.246761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1217.296618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1217.333311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1217.379668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1217.427879] LNetError: 68207:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1218.274471] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1218.277102] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1220.320079] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1221.178431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1224.416497] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1224.762590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1224.801778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1224.901243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1224.945054] LNet: 68314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1224.961466] LNetError: 68314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1224.968376] LNet: Removed LNI 192.168.203.32@tcp [ 1224.999866] LNet: Removed LNI 192.168.203.32@tcp1 [ 1225.073362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1225.141320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1225.160046] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1225.725340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1225.750924] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1226.559702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1226.598924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.679447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1226.716406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1226.743239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1226.798600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1226.836316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1226.871192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1226.886600] LNetError: 68586:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1227.168671] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1229.419910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1233.313664] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1233.321210] LNet: Skipped 1 previous similar message [ 1238.518337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1238.562139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.630410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.824907] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:08:22 (1789132102) [ 1246.072722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1246.097511] LNet: 69303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.121728] LNetError: 69303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.131331] LNetError: 69303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1246.140567] LNet: Removed LNI 192.168.203.32@tcp [ 1246.153120] LNet: Removed LNI 192.168.203.32@tcp1 [ 1246.190119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.268664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1246.307608] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1246.978641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1247.009626] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1247.871747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1247.954274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1247.995266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1248.025976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1248.054848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1248.105706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1248.142719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 0 [ 1248.230368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1248.287069] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.32@tcp [ 1249.311788] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1249.317274] LNet: Skipped 1 previous similar message [ 1264.694617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.880343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.928854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1264.982895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 500 [ 1265.122132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1272.949856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1273.024790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1273.040183] LNet: 69811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.052586] LNetError: 69811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.062405] LNetError: 69811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1273.070201] LNet: Removed LNI 192.168.203.32@tcp [ 1273.084439] LNet: Removed LNI 192.168.203.32@tcp1 [ 1273.105636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.156517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1273.188631] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1274.219860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1274.267420] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1275.420219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1275.475443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.545642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1275.593381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1275.648978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1275.684245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1275.721777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1275.765284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.32@tcp --health 0 [ 1276.191462] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1276.196438] LNet: Skipped 3 previous similar messages [ 1277.077477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1287.440204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1287.490937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1287.596526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1294.219410] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:09:11 (1789132151) [ 1294.579551] LNet: 70857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.583964] LNetError: 70857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.589233] LNetError: 70857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1294.594773] LNet: Removed LNI 192.168.203.32@tcp [ 1294.619129] LNet: Removed LNI 192.168.203.32@tcp1 [ 1295.230892] Key type .llcrypt unregistered [ 1295.234757] Key type ._llcrypt unregistered [ 1301.357686] Key type ._llcrypt registered [ 1301.359370] Key type .llcrypt registered [ 1301.570597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1301.588364] alg: No test for adler32 (adler32-zlib) [ 1302.527572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1302.564531] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1311.502468] Lustre: DEBUG MARKER: Initial discovery [ 1311.577600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1313.891800] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1314.003728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1314.045827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1314.079425] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1314.104526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1314.147924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.132@tcp -r 1 -m GET -l 3 [ 1314.193321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.132@tcp -r 1 -m GET -e local_error [ 1314.231744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.132@tcp -r 1 -m PUT -l 6 [ 1314.848406] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1315.779622] Lustre: DEBUG MARKER: Force oleg332-client.virtnet to discover 192.168.203.132@tcp (in background) [ 1315.853684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.132@tcp [ 1317.252607] Lustre: DEBUG MARKER: Force oleg332-server to discover 192.168.203.32@tcp [ 1318.947180] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1319.669042] Lustre: DEBUG MARKER: Wait for 71614 [ 1321.122598] Lustre: DEBUG MARKER: Finished wait on 71614 [ 1321.205254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1321.228368] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1321.235326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1321.608604] LNet: 72076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1321.624489] LNetError: 72076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.661935] LNet: Removed LNI 192.168.203.32@tcp [ 1323.018606] Key type .llcrypt unregistered [ 1323.024292] Key type ._llcrypt unregistered [ 1339.192666] Key type ._llcrypt registered [ 1339.204174] Key type .llcrypt registered [ 1340.470822] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:09:57 (1789132197) [ 1340.974686] Key type .llcrypt unregistered [ 1340.977595] Key type ._llcrypt unregistered [ 1347.226455] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1347.396941] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1347.406842] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1347.410878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1349.900990] Key type ._llcrypt registered [ 1349.904131] Key type .llcrypt registered [ 1350.061907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1350.084371] alg: No test for adler32 (adler32-zlib) [ 1350.991248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.032496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1351.050805] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1351.570273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1351.585309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1359.250972] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:10:16 (1789132216) [ 1359.342945] LNet: 74164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.355419] LNetError: 74164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1359.365659] LNet: Removed LNI 10.1.2.3@tcp [ 1359.379271] LNet: Removed LNI 192.168.203.32@tcp [ 1360.163763] Key type .llcrypt unregistered [ 1360.166774] Key type ._llcrypt unregistered [ 1365.993545] Key type ._llcrypt registered [ 1365.995379] Key type .llcrypt registered [ 1366.184704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1366.207265] alg: No test for adler32 (adler32-zlib) [ 1367.106561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.157697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1367.188130] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1367.879698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1367.895586] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1368.946111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.32@tcp [ 1370.019810] LNetError: Unexpected error -22 connecting to 192.168.203.32@tcp at host 192.168.203.32:988 [ 1375.183163] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:10:32 (1789132232) [ 1375.313699] LNet: 75298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1375.331972] LNetError: 75298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1375.340835] LNet: Removed LNI 10.1.2.3@tcp [ 1375.354210] LNet: Removed LNI 192.168.203.32@tcp [ 1376.131622] Key type .llcrypt unregistered [ 1376.134271] Key type ._llcrypt unregistered [ 1382.104774] Key type ._llcrypt registered [ 1382.110197] Key type .llcrypt registered [ 1382.396898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1382.412116] alg: No test for adler32 (adler32-zlib) [ 1383.298217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1383.341861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1383.357930] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1384.019421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1384.041355] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 1384.869144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.32@tcp1 --nid 192.168.203.32@tcp2 [ 1388.907398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp1 192.168.203.32@tcp2 [ 1388.926341] LNetError: 76145:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.32@tcp1 to 192.168.203.32@tcp2 - Route table may be misconfigured [ 1388.957539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp2 192.168.203.32@tcp1 [ 1394.956213] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:10:52 (1789132252) [ 1395.057832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.094934] LNet: 76711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.101150] LNetError: 76711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.112761] LNet: Removed LNI 192.168.203.32@tcp1 [ 1396.136784] LNet: Removed LNI 192.168.203.32@tcp2 [ 1396.193469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.267520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.302933] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1397.109367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1397.130778] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1397.996343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1398.075026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.32@tcp [ 1398.110912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp1 192.168.203.32@tcp1 [ 1398.172979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.32@tcp -d 192.168.203.32@tcp -e network_timeout [ 1398.219507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -e network_timeout [ 1398.260271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -e network_timeout [ 1398.308357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -e network_timeout [ 1398.354563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1398.372556] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1398.390370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1398.518393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1398.619531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1399.327259] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1404.111835] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:11:01 (1789132261) [ 1404.218353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1404.249828] LNet: 77575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.267831] LNetError: 77575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.277425] LNetError: 77575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1404.286545] LNet: Removed LNI 192.168.203.32@tcp [ 1405.352664] LNet: Removed LNI 192.168.203.32@tcp1 [ 1405.385810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.466256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1405.696630] LNet: 77621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.013559] Key type .llcrypt unregistered [ 1406.017761] Key type ._llcrypt unregistered [ 1416.567631] Key type ._llcrypt registered [ 1416.569493] Key type .llcrypt registered [ 1417.845883] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:11:14 (1789132274) [ 1418.384753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1418.403732] alg: No test for adler32 (adler32-zlib) [ 1419.284987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.531416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1422.062341] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:11:19 (1789132279) [ 1422.171638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.204158] LNet: 78715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.229414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.293187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.313330] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1422.892354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1422.909677] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1423.714326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1423.751107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp1 [ 1423.805344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp1 [ 1429.479366] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:11:26 (1789132286) [ 1430.991138] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg332-server [ 1432.652131] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:11:29 (1789132289) [ 1434.505819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg332-server [ 1436.260206] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:11:33 (1789132293) [ 1437.895166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg332-server [ 1439.498668] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:11:36 (1789132296) [ 1441.281472] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg332-server [ 1442.867485] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:11:39 (1789132299) [ 1444.675420] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg332-server [ 1446.239184] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:11:43 (1789132303) [ 1447.992581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg332-server [ 1449.728226] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:11:46 (1789132306) [ 1451.277517] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg332-server [ 1452.612569] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:11:49 (1789132309) [ 1454.327275] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg332-server [ 1455.724273] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:11:52 (1789132312) [ 1457.198823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg332-server [ 1458.538521] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:11:55 (1789132315) [ 1458.662583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.690798] LNet: 82596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.698550] LNetError: 82596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.703869] LNet: Removed LNI 192.168.203.32@tcp [ 1459.755242] LNet: Removed LNI 192.168.203.32@tcp1 [ 1459.788654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.845327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.890207] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1460.433075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1460.494456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1460.626879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.647827] LNet: 82743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.654753] LNetError: 82743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1460.659228] LNetError: 82743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1460.672472] LNet: Removed LNI 192.168.203.32@tcp [ 1460.693793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1460.725678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1460.746301] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1461.303143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1461.459350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1462.504249] LNet: Removed LNI 192.168.203.32@tcp [ 1462.537433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.619470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1462.644304] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1463.345733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1463.429769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1464.821546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.847721] LNet: 83045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.853715] LNet: 83045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1464.867173] LNetError: 83045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.876882] LNetError: 83045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1465.893582] LNet: Removed LNI 192.168.203.32@tcp [ 1465.912888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1465.979722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1466.007763] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1466.708874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1466.942864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.976803] LNetError: 83190:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.011259] LNet: Removed LNI 192.168.203.32@tcp [ 1468.040377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.152444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.183514] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1468.756429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1468.848297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1469.072702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.101961] LNet: 83338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.109621] LNet: 83338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1470.182052] LNet: Removed LNI 192.168.203.32@tcp [ 1470.231757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1470.303546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1470.341843] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1471.218907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1471.412289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.437070] LNetError: 83483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.445625] LNetError: 83483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1471.477169] LNet: Removed LNI 192.168.203.32@tcp [ 1471.514190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.596595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1471.647521] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1472.320909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1472.400962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1472.653604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.768695] LNet: Removed LNI 192.168.203.32@tcp [ 1473.801670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.852848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1473.879895] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1474.520963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1474.806981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1475.877353] LNet: Removed LNI 192.168.203.32@tcp [ 1475.921450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1476.031633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1476.069084] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1476.891744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1476.951201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1477.168707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.198491] LNet: 83923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.202892] LNet: 83923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1478.250276] LNet: Removed LNI 192.168.203.32@tcp [ 1478.292605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1478.342540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1478.381259] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1479.301751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1479.600184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.642113] LNetError: 84068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1479.653288] LNetError: 84068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1479.686358] LNet: Removed LNI 192.168.203.32@tcp [ 1479.743514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.814548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.860851] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1480.765463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1480.834759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1481.043685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.087167] LNet: Removed LNI 192.168.203.32@tcp [ 1482.130668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1482.230376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1482.274736] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1482.999243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1483.237863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.327455] LNet: Removed LNI 192.168.203.32@tcp [ 1484.356236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.402153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.416328] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1485.115260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1485.172699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.194182] LNet: Removed LNI 192.168.203.32@tcp [ 1485.213650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.283413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1485.299621] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1486.053533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1486.156943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.216393] LNet: Removed LNI 192.168.203.32@tcp [ 1486.245555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.319936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1486.412974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.455361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.493281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1486.546939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.575514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.642310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1486.721307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1486.769016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.833405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1486.868954] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1487.511737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1487.573606] LNet: Removed LNI 192.168.203.32@tcp [ 1487.618577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1487.707551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1493.709332] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:12:30 (1789132350) [ 1493.936759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1493.974663] LNet: 85379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.992719] LNet: 85379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1494.025618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1494.109640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1494.167723] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1495.202966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.251069] LNet: Removed LNI 192.168.203.32@tcp231 [ 1495.269422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1495.374742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1495.399864] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1495.410360] LNet: 85527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1495.805558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1495.851757] LNet: Removed LNI 192.168.203.32@tcp231 [ 1495.866985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1495.887869] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1496.794033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1496.811327] LNetError: 85672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1496.813981] LNetError: 85672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1496.830057] LNet: Removed LNI 192.168.203.32@tcp231 [ 1496.848525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.912956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1496.942388] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/0] [ 1496.947427] LNet: 85680:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1497.242403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1497.266788] LNet: Removed LNI 192.168.203.32@tcp231 [ 1497.281844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1497.308362] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/0] [ 1498.454989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1498.501168] LNet: Removed LNI 192.168.203.32@tcp231 [ 1498.538654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1498.624923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1498.663025] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/60] [ 1498.669647] LNet: 85831:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1499.118794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1499.149805] LNet: Removed LNI 192.168.203.32@tcp231 [ 1499.162312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1499.195171] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/60] [ 1500.051575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.075344] LNet: Removed LNI 192.168.203.32@tcp231 [ 1500.089638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.172772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1500.203784] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1500.712446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1500.756337] LNet: Removed LNI 192.168.203.32@tcp231 [ 1500.773612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1500.807037] LNet: Added LNI 192.168.203.32@tcp231 [8/256/0/180] [ 1507.517470] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:12:44 (1789132364) [ 1507.666582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.710319] LNet: Removed LNI 192.168.203.32@tcp231 [ 1507.750487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.827468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1507.856995] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1508.667698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1508.733147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1508.769250] LNet: Removed LNI 192.168.203.32@tcp232 [ 1508.793731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1508.826193] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1509.605568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1509.657560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1509.699320] LNet: Removed LNI 192.168.203.32@tcp232 [ 1509.718553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1509.742562] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1510.557749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1510.617636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1510.664704] LNet: Removed LNI 192.168.203.32@tcp232 [ 1510.691203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1510.727868] LNet: Added LNI 192.168.203.32@tcp232 [8/256/0/180] [ 1511.964950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1512.012561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1512.062969] LNet: Removed LNI 192.168.203.32@tcp232 [ 1512.107448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1519.109627] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:12:56 (1789132376) [ 1520.746255] Key type .llcrypt unregistered [ 1520.748807] Key type ._llcrypt unregistered [ 1527.672342] Key type ._llcrypt registered [ 1527.675656] Key type .llcrypt registered [ 1527.950211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.972256] alg: No test for adler32 (adler32-zlib) [ 1529.002907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1529.026760] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1536.336725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1545.042957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.115383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1545.157540] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1546.737768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_interrupt -r 2 [ 1546.771155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 2 [ 1546.798759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_interrupt -r 2 [ 1546.830770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 2 [ 1546.862249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_interrupt -r 2 [ 1546.911373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_interrupt -r 2 [ 1546.955272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_interrupt -r 2 [ 1547.000976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_interrupt -r 2 [ 1547.064855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1547.081318] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1547.085731] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1547.382255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1547.422891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1547.459610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1547.507129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1547.555105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1547.603868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1547.652558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1547.689497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1547.712550] LNet: 88486:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1547.745796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1547.789491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1547.841982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1547.884599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1547.937866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1547.979801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1548.641096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_dropped -r 2 [ 1548.686916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 2 [ 1548.757847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_dropped -r 2 [ 1548.802326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 2 [ 1548.834377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_dropped -r 2 [ 1548.875950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_dropped -r 2 [ 1548.914119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_dropped -r 2 [ 1548.960950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_dropped -r 2 [ 1549.016593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1549.061214] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1549.331954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1549.365917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1549.400119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1549.426305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1549.469389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1549.507543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1549.531743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1549.557661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1549.574260] LNet: 88606:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1549.610064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1549.653694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1549.679893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1549.715347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1549.740660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1549.774343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1550.243243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_aborted -r 2 [ 1550.294259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 2 [ 1550.321594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_aborted -r 2 [ 1550.365261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 2 [ 1550.401305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_aborted -r 2 [ 1550.430662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_aborted -r 2 [ 1550.482379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_aborted -r 2 [ 1550.521177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_aborted -r 2 [ 1550.549324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1550.576714] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1550.767773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1550.799790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1550.829551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1550.891959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1550.931604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1550.965640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1551.008551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1551.048800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1551.070355] LNet: 88726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1551.097523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1551.127196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1551.158505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1551.191949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1551.217978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1551.248774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1551.778947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_no_route -r 2 [ 1551.817281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 2 [ 1551.856292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_no_route -r 2 [ 1551.892305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 2 [ 1551.925486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_no_route -r 2 [ 1551.969252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_no_route -r 2 [ 1552.016726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_no_route -r 2 [ 1552.083762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_no_route -r 2 [ 1552.149480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1552.441667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1552.488579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1552.572953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1552.614980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1552.643580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1552.698104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1552.737796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1552.770929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1552.803193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1552.852912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1552.891671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1552.919327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1552.977178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1553.025138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1553.581435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e local_timeout -r 2 [ 1553.615719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 2 [ 1553.651896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e local_timeout -r 2 [ 1553.686704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 2 [ 1553.718181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e local_timeout -r 2 [ 1553.746704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e local_timeout -r 2 [ 1553.775685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e local_timeout -r 2 [ 1553.827560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e local_timeout -r 2 [ 1553.880827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1553.915476] LNet: There was an unexpected network error while writing to 192.168.203.132: rc = -22 [ 1553.933742] LNet: Skipped 1 previous similar message [ 1554.257382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1554.294228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1554.330993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1554.372182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1554.422125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1554.456090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1554.493476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1554.546365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1554.574912] LNet: 88967:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1554.584682] LNet: 88967:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1554.604453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1554.642942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1554.689727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1554.718099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1554.751729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1554.804937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1555.413562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp -m GET -e remote_dropped -r 2 [ 1555.501549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 2 [ 1555.572815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp -m GET -e remote_dropped -r 2 [ 1555.625325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 2 [ 1555.683903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp -m GET -e remote_dropped -r 2 [ 1555.742759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.32@tcp1 -d 192.168.203.132@tcp1 -m GET -e remote_dropped -r 2 [ 1555.791463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp -m GET -e remote_dropped -r 2 [ 1555.855107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.32@tcp1 -d 192.168.203.32@tcp1 -m GET -e remote_dropped -r 2 [ 1555.893385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.32@tcp 192.168.203.132@tcp [ 1555.917830] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 1556.192922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1556.259497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1556.303397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1556.344464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1556.386437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1556.423448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1556.463985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1556.491958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1556.518394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1556.554942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1556.606388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1556.677318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1556.717537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1556.757642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1562.294372] LNet: 89165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.304964] LNetError: 89165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.316339] LNet: Removed LNI 192.168.203.32@tcp [ 1562.333881] LNet: Removed LNI 192.168.203.32@tcp1 [ 1562.769854] Key type .llcrypt unregistered [ 1562.772281] Key type ._llcrypt unregistered [ 1576.300805] Key type ._llcrypt registered [ 1576.302532] Key type .llcrypt registered [ 1577.466320] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:13:54 (1789132434) [ 1577.817285] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1577.831387] alg: No test for adler32 (adler32-zlib) [ 1578.755480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.797709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.821693] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1579.772843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1579.810486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.32@tcp [ 1579.838533] LNetError: 90151:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1579.929203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.32@tcp [ 1580.051287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1587.632835] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:14:04 (1789132444) [ 1587.779240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.817847] LNet: 90736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1587.828947] LNetError: 90736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.853531] LNet: Removed LNI 192.168.203.32@tcp [ 1587.906336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1588.000905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1588.021035] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1588.970462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1588.995364] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1590.260661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.32@tcp [ 1590.281963] LNetError: 91007:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.32@tcp to recovery [ 1591.007766] LNet: 1 local NIs in recovery (showing 1): 192.168.203.32@tcp [ 1591.009823] LNet: There was an unexpected network error while writing to 192.168.203.32: rc = -22 [ 1591.535992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.32@tcp [ 1599.492837] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:14:16 (1789132456) [ 1601.225916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg332-server [ 1602.784315] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 09:14:19 (1789132459) [ 1604.258122] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg332-server [ 1605.976821] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 09:14:23 (1789132463) [ 1607.680053] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg332-server [ 1609.284315] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:14:26 (1789132466) [ 1609.540495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.563210] LNet: 92636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.573182] LNetError: 92636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.594247] LNet: Removed LNI 192.168.203.32@tcp [ 1610.598065] LNetError: 92636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1610.628900] LNet: Removed LNI 192.168.203.32@tcp1 [ 1610.707060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.763315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.788855] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1611.495485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1611.546566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1611.603848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1611.683724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1611.720869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1611.756900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1616.724811] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:14:34 (1789132474) [ 1616.933664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.964406] LNet: 93361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.972943] LNetError: 93361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.991637] LNet: Removed LNI 192.168.203.32@tcp [ 1617.035060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.104816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.131044] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1623.604747] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:14:40 (1789132480) [ 1623.780652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.817027] LNet: 94061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.828221] LNetError: 94061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.853192] LNet: Removed LNI 192.168.203.32@tcp [ 1623.895239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.007548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1624.081192] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1630.683079] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:14:47 (1789132487) [ 1632.099324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1633.891744] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:14:50 (1789132490) [ 1634.257635] LNet: 95099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1634.265431] LNetError: 95099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1634.279981] LNet: Removed LNI 192.168.203.32@tcp [ 1634.966820] Key type .llcrypt unregistered [ 1634.971072] Key type ._llcrypt unregistered [ 1641.326952] Key type ._llcrypt registered [ 1641.332158] Key type .llcrypt registered [ 1641.506748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1641.522993] alg: No test for adler32 (adler32-zlib) [ 1642.517614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1642.569377] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1649.234344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1653.192395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.132@tcp [ 1653.478145] LNet: 95524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.482455] LNetError: 95524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.493991] LNet: Removed LNI 192.168.203.32@tcp [ 1653.787561] Key type .llcrypt unregistered [ 1653.791556] Key type ._llcrypt unregistered [ 1663.909761] Key type ._llcrypt registered [ 1663.915064] Key type .llcrypt registered [ 1665.263222] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1666.538987] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1668.102534] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:15:25 (1789132525) [ 1669.606193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1671.307352] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:15:28 (1789132528) [ 1673.357569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg332-server [ 1675.292957] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:15:32 (1789132532) [ 1677.152659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg332-server [ 1678.727944] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:15:36 (1789132536) [ 1679.806865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1679.968295] alg: No test for adler32 (adler32-zlib) [ 1681.077108] Lustre: Lustre: Build Version: 2.17.57_132_gd76390e [ 1681.344619] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1683.017861] Key type lgssc registered [ 1684.284065] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1705.039151] Key type lgssc unregistered [ 1705.315625] LNet: 98710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.323306] LNetError: 98710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.347786] LNet: Removed LNI 192.168.203.32@tcp [ 1706.164647] Key type .llcrypt unregistered [ 1706.166876] Key type ._llcrypt unregistered [ 1721.029579] Key type ._llcrypt registered [ 1721.033502] Key type .llcrypt registered [ 1722.368197] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:16:19 (1789132579) [ 1722.733467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1722.746385] alg: No test for adler32 (adler32-zlib) [ 1723.640729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.739667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1723.809661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1723.874311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1723.919790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1730.375309] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:16:27 (1789132587) [ 1730.526471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.574968] LNet: 100124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.616593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.677027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1730.724457] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1731.718602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1731.738636] LNet: 100268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.744896] LNetError: 100268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.778640] LNet: Removed LNI 192.168.203.32@tcp [ 1731.802937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1731.875968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1731.902850] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1732.546943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.574218] LNetError: 100297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.593845] LNet: Removed LNI 192.168.203.32@tcp [ 1732.630540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.698635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1732.748722] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1739.449440] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:16:36 (1789132596) [ 1739.564740] LNet: 100881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.573856] LNet: 100881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1739.581575] LNetError: 100881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.611402] LNet: Removed LNI 192.168.203.32@tcp [ 1740.287737] Key type .llcrypt unregistered [ 1740.289993] Key type ._llcrypt unregistered [ 1746.093611] Key type ._llcrypt registered [ 1746.096862] Key type .llcrypt registered [ 1746.309841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1746.326943] alg: No test for adler32 (adler32-zlib) [ 1747.213145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.288193] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1747.292542] LNet: *** cfs_fail_loc=e003, val=100*** [ 1748.068520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1748.094607] LNet: *** cfs_fail_loc=e003, val=100*** [ 1748.097015] LNet: Skipped 2 previous similar messages [ 1748.103770] Ping reply received with mapping: [ 1748.106347] found metadata reply NI[192.168.203.32@tcp] [ 1748.109967] 20 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1748.112930] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1748.116093] 5a 5a 5a 5a 5a 5a 5a 5a [ 1748.214944] LNet: 101289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1748.222340] LNetError: 101289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.233763] LNet: Removed LNI 192.168.203.32@tcp [ 1748.804886] Key type .llcrypt unregistered [ 1748.808445] Key type ._llcrypt unregistered [ 1759.807217] Key type ._llcrypt registered [ 1759.809469] Key type .llcrypt registered [ 1761.065186] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:16:58 (1789132618) [ 1761.614136] Key type .llcrypt unregistered [ 1761.622073] Key type ._llcrypt unregistered [ 1768.308674] Key type ._llcrypt registered [ 1768.312524] Key type .llcrypt registered [ 1768.505673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1768.520516] alg: No test for adler32 (adler32-zlib) [ 1769.706811] LNetError: 102346:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1770.189845] Key type .llcrypt unregistered [ 1770.194406] Key type ._llcrypt unregistered [ 1774.685260] Key type ._llcrypt registered [ 1774.688366] Key type .llcrypt registered [ 1775.966533] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:17:12 (1789132632) [ 1776.433191] Key type .llcrypt unregistered [ 1776.436238] Key type ._llcrypt unregistered [ 1782.694205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1782.703452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1785.308726] Key type ._llcrypt registered [ 1785.311915] Key type .llcrypt registered [ 1785.537974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.558639] alg: No test for adler32 (adler32-zlib) [ 1786.603017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1786.692327] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1796.090313] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:17:33 (1789132653) [ 1796.295515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1796.325043] LNet: 104130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.334393] LNetError: 104130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1796.361443] LNet: Removed LNI 10.1.2.3@tcp [ 1796.401062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.132454] Key type .llcrypt unregistered [ 1797.134227] Key type ._llcrypt unregistered [ 1804.082424] Key type ._llcrypt registered [ 1804.085590] Key type .llcrypt registered [ 1804.290275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1804.326319] alg: No test for adler32 (adler32-zlib) [ 1805.256986] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1805.305280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1805.460983] LNet: 104426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1805.469419] LNetError: 104426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.489585] LNet: Removed LNI 192.168.203.32@tcp [ 1806.162964] Key type .llcrypt unregistered [ 1806.169190] Key type ._llcrypt unregistered [ 1817.579224] Key type ._llcrypt registered [ 1817.585098] Key type .llcrypt registered [ 1819.029539] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:17:55 (1789132675) [ 1819.783955] Key type .llcrypt unregistered [ 1819.786526] Key type ._llcrypt unregistered [ 1826.334118] Key type ._llcrypt registered [ 1826.336815] Key type .llcrypt registered [ 1826.512656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1826.532067] alg: No test for adler32 (adler32-zlib) [ 1833.523806] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:18:10 (1789132690) [ 1833.687172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.718316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.790186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.818527] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1834.520924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1834.569801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1834.598409] LNet: Removed LNI 192.168.203.32@tcp [ 1834.890931] LNet: 106182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.314642] Key type .llcrypt unregistered [ 1835.318117] Key type ._llcrypt unregistered [ 1846.603473] Key type ._llcrypt registered [ 1846.606666] Key type .llcrypt registered [ 1848.020153] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:18:24 (1789132704) [ 1848.271306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.291813] alg: No test for adler32 (adler32-zlib) [ 1849.206368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.255943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1849.286683] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1849.944993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1849.988522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.32@tcp [ 1850.007878] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1850.012113] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1850.015949] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1849, 0, 0 [ 1850.021522] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1850.025280] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1850.029746] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1850.034358] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1849, 0, 0 [ 1850.040124] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1850.043449] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1850.048385] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1850.051964] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1849, 0, 1899 [ 1850.058102] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1850.061894] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1850.066036] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1850.070303] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1849, 0, 1899 [ 1850.075877] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1855.445457] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:18:32 (1789132712) [ 1855.864820] LNet: 107721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1855.872248] LNetError: 107721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1855.888947] LNet: Removed LNI 192.168.203.32@tcp [ 1856.645460] Key type .llcrypt unregistered [ 1856.647445] Key type ._llcrypt unregistered [ 1863.215449] Key type ._llcrypt registered [ 1863.218925] Key type .llcrypt registered [ 1863.496817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1863.523475] alg: No test for adler32 (adler32-zlib) [ 1864.452164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1864.477849] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1872.943905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 1882.248714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.295632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1882.327960] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 1883.209848] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1883.397369] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1883.407139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1883.418574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1885.799468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1885.822285] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1895.314747] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:19:12 (1789132752) [ 1895.515343] LNet: 109233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.529251] LNetError: 109233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.535455] LNet: Removed LNI 192.168.203.32@tcp [ 1896.543181] LNet: Removed LNI 192.168.203.32@tcp1 [ 1896.545527] LNetError: 109233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1896.552976] LNetError: 109233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1896.570493] LNet: Removed LNI 10.1.2.3@tcp99 [ 1897.312731] Key type .llcrypt unregistered [ 1897.316039] Key type ._llcrypt unregistered [ 1903.711792] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1903.725152] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1906.299855] Key type ._llcrypt registered [ 1906.302226] Key type .llcrypt registered [ 1906.444836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1906.456428] alg: No test for adler32 (adler32-zlib) [ 1907.343775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.415945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.458591] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1908.112713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1908.127491] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1910.000843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1910.055642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1910.101771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1910.124344] LNetError: 109982:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1910.155670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1910.197715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1910.661583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1910.773301] LNet: 110002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.781555] LNetError: 110002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1910.792238] LNet: Removed LNI 10.1.2.3@tcp [ 1911.846880] LNet: Removed LNI 192.168.203.32@tcp [ 1912.523782] Key type .llcrypt unregistered [ 1912.534702] Key type ._llcrypt unregistered [ 1923.354177] Key type ._llcrypt registered [ 1923.356297] Key type .llcrypt registered [ 1924.501356] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:19:41 (1789132781) [ 1925.018455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1925.037963] alg: No test for adler32 (adler32-zlib) [ 1925.960506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.019191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1926.060892] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1926.630685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 1926.735094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg332-client@tcp [ 1932.173149] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:19:49 (1789132789) [ 1932.325106] LNet: 111534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.337137] LNetError: 111534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.350065] LNet: Removed LNI 192.168.203.32@tcp [ 1932.994435] Key type .llcrypt unregistered [ 1932.999860] Key type ._llcrypt unregistered [ 1938.720189] Key type ._llcrypt registered [ 1938.729640] Key type .llcrypt registered [ 1938.861448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.884906] alg: No test for adler32 (adler32-zlib) [ 1939.743246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1939.759364] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1939.801698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1945.543205] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:20:02 (1789132802) [ 1946.932955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1948.608522] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:20:05 (1789132805) [ 1950.137709] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1952.261185] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:20:08 (1789132808) [ 1952.404738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1952.467304] LNet: 112954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.481521] LNetError: 112954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.512083] LNet: Removed LNI 192.168.203.32@tcp [ 1952.547928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.714482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.732986] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1953.606946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1953.651859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1961.461110] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:20:18 (1789132818) [ 1961.577862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.601769] LNet: 113665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.609851] LNetError: 113665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.642885] LNet: Removed LNI 192.168.203.32@tcp [ 1961.677867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1961.750385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1961.790278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1962.560809] Key type .llcrypt unregistered [ 1962.564218] Key type ._llcrypt unregistered [ 1974.834894] Key type ._llcrypt registered [ 1974.837171] Key type .llcrypt registered [ 1976.290276] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:20:33 (1789132833) [ 1976.724529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.741679] alg: No test for adler32 (adler32-zlib) [ 1977.656448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.728705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.782838] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1978.636640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1978.731437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 1985.269369] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:20:42 (1789132842) [ 1985.409746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.430914] LNet: 115224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1985.439117] LNetError: 115224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.464638] LNet: Removed LNI 192.168.203.32@tcp [ 1985.499927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.564319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1985.612850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1990.947980] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:20:48 (1789132848) [ 1991.118650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.151026] LNet: 115809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.192617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.258669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1991.579894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1991.606255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1991.666351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1992.101977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1992.137365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1992.233212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1992.259299] LNet: 115842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.299543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.373647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1992.805624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1992.849563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1992.916813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1993.375977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1993.407195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1998.901585] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:20:56 (1789132856) [ 1999.019577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1999.036704] LNet: 116442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1999.055643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1999.086372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1999.109741] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 1999.788487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1999.809880] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2000.466750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2000.523597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2000.721929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.739772] LNetError: 116721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.744220] LNet: Removed LNI 192.168.203.32@tcp [ 2000.755542] LNet: Removed LNI 192.168.203.32@tcp2 [ 2000.790181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.852989] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2000.860339] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2000.864205] LNet: 116727:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2001.298555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2001.513959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.591205] LNet: Removed LNI 192.168.203.32@tcp [ 2002.594635] LNetError: 116764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.603880] LNetError: 116764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2002.621674] LNet: Removed LNI 192.168.203.32@tcp2 [ 2002.642500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.681396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.697060] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2003.345130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2003.366370] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2004.080577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2004.136584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2004.221621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2004.427811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2004.439675] LNet: 117049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2004.445902] LNet: 117049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2005.471551] LNet: Removed LNI 192.168.203.32@tcp [ 2005.479243] LNetError: 117049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.489260] LNetError: 117049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2005.518461] LNet: Removed LNI 192.168.203.32@tcp2 [ 2005.574936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.659599] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2005.677466] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2005.684874] LNet: 117055:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2006.202634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2006.456697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.491987] LNet: Removed LNI 192.168.203.32@tcp [ 2007.524776] LNet: Removed LNI 192.168.203.32@tcp2 [ 2007.575535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2007.631592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2007.676070] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2008.655397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2008.696429] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2009.562924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2009.641309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2009.867849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2009.933564] LNetError: 117372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.949358] LNetError: 117372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2009.964573] LNet: Removed LNI 192.168.203.32@tcp [ 2009.979201] LNet: Removed LNI 192.168.203.32@tcp2 [ 2010.005501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2010.146243] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2010.169428] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2010.186871] LNet: 117378:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2011.046087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2011.351458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.383258] LNet: Removed LNI 192.168.203.32@tcp [ 2012.402814] LNet: Removed LNI 192.168.203.32@tcp2 [ 2012.442655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.581265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2012.610832] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2013.390171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2013.427928] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2014.585751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2014.674091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2014.792529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2015.130679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.155131] LNet: 117699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.169537] LNet: 117699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2016.223187] LNet: Removed LNI 192.168.203.32@tcp [ 2017.253567] LNet: Removed LNI 192.168.203.32@tcp2 [ 2017.295166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.371258] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2017.386737] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2017.393318] LNet: 117706:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2017.922174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2018.197207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2018.221929] LNetError: 117743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.226455] LNetError: 117743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2018.233528] LNet: Removed LNI 192.168.203.32@tcp [ 2019.312247] LNet: Removed LNI 192.168.203.32@tcp2 [ 2019.336802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2019.392115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2019.416343] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2019.931233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2019.955437] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2020.751517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2020.800175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2020.882913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2021.347720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2022.431880] LNet: Removed LNI 192.168.203.32@tcp [ 2022.449305] LNet: Removed LNI 192.168.203.32@tcp2 [ 2022.476431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.552907] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2022.567712] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2022.578891] LNet: 118045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2023.194770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2023.584795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2024.672034] LNet: Removed LNI 192.168.203.32@tcp [ 2024.680600] LNet: Removed LNI 192.168.203.32@tcp2 [ 2024.723590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2024.792194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.812889] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2025.425562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2025.446881] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2026.590570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2026.625617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2026.694593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2026.767607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2027.127582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2028.191439] LNet: Removed LNI 192.168.203.32@tcp [ 2029.224040] LNet: Removed LNI 192.168.203.32@tcp2 [ 2029.258627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.343464] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2029.359445] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2030.003880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2036.845485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2038.270378] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:21:35 (1789132895) [ 2038.373759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.416512] LNet: 119161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.424779] LNet: 119161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2038.439043] LNetError: 119161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.445633] LNetError: 119161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2038.451735] LNet: Removed LNI 192.168.203.32@tcp [ 2039.463858] LNet: Removed LNI 192.168.203.32@tcp2 [ 2039.485234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.764847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2039.790849] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2040.344435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2040.359453] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2041.161866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2041.184826] LNet: Removed LNI 10.1.2.3@tcp [ 2041.199190] LNet: Removed LNI 192.168.203.32@tcp [ 2041.228612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2041.278771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2041.303188] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2041.307821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2041.705511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2041.728227] LNet: Removed LNI 192.168.203.32@tcp [ 2041.739847] LNet: Removed LNI 10.1.2.3@tcp [ 2047.334907] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 09:21:44 (1789132904) [ 2047.444742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2047.503331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.587089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.623539] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2048.399397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2048.459504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.505542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.554936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.593735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.631595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.678133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.713866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.754363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.796311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2048.846848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.32@tcp [ 2049.076120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2055.094320] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:21:52 (1789132912) [ 2055.229779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2056.298346] LNet: Removed LNI 192.168.203.32@tcp [ 2056.333131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.426527] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2056.596598] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2056.602075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2056.605762] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2058.957874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.993721] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2059.700199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2059.724124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2061.786055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2061.859968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2061.954490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.32@tcp -m PUT -e local_timeout -r 1 [ 2062.036249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2062.112230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2062.119364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2062.475319] LNetError: 120844:0:(lib-move.c:4801:lnet_parse()) 192.168.203.32@tcp, src 192.168.203.32@tcp: Dropping PUT (error -108 looking up sender) [ 2064.543191] LNet: Removed LNI 10.1.2.3@tcp [ 2065.574622] LNet: Removed LNI 192.168.203.32@tcp [ 2065.926843] Key type .llcrypt unregistered [ 2065.928909] Key type ._llcrypt unregistered [ 2078.082590] Key type ._llcrypt registered [ 2078.088686] Key type .llcrypt registered [ 2079.591608] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:22:16 (1789132936) [ 2079.978467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2079.992951] alg: No test for adler32 (adler32-zlib) [ 2080.963475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2081.418250] LNet: 122160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2082.085430] Key type .llcrypt unregistered [ 2082.093042] Key type ._llcrypt unregistered [ 2088.412400] Key type ._llcrypt registered [ 2088.415934] Key type .llcrypt registered [ 2088.605484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2088.629562] alg: No test for adler32 (adler32-zlib) [ 2089.541457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2089.560613] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2096.919774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 2097.112170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2097.636074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.132@tcp [ 2097.777782] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.132@tcp [ 2097.901562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2097.936662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2097.966982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2097.996832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2098.039256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2098.081711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2098.117586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2098.157801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2098.182435] LNet: 122614:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2098.204707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2098.276571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2098.319908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2098.371553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2098.402868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2098.443873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2099.121526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2099.418324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2099.452583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2099.479570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2099.510431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2099.546397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2099.585872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2099.617521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2099.658255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2099.687070] LNet: 122731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2099.732762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2099.798737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2099.851137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2099.896414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2099.928145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2099.957471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2100.596646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.132@tcp [ 2100.866962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2100.924526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2100.949124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2100.976799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2101.005867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2101.034456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2101.077818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2101.120182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2101.152349] LNet: 122848:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2101.186100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2101.228741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2101.274391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2101.308742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2101.344596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2101.387795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2102.021380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2102.240930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2102.264348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2102.288535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2102.319573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2102.345425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2102.370793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2102.395194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2102.420252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2102.441805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2102.464567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2102.487995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2102.512557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2102.541129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2102.566635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2102.852463] LNet: 123015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2102.859686] LNetError: 123015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.879633] LNet: Removed LNI 192.168.203.32@tcp [ 2103.258922] Key type .llcrypt unregistered [ 2103.263217] Key type ._llcrypt unregistered [ 2117.965502] Key type ._llcrypt registered [ 2117.968609] Key type .llcrypt registered [ 2119.185935] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:22:56 (1789132976) [ 2119.725540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2119.747907] alg: No test for adler32 (adler32-zlib) [ 2120.786418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.258566] LNet: 123903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.023393] Key type .llcrypt unregistered [ 2122.029356] Key type ._llcrypt unregistered [ 2130.687165] Key type ._llcrypt registered [ 2130.688583] Key type .llcrypt registered [ 2131.191602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2131.243698] alg: No test for adler32 (adler32-zlib) [ 2132.153991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2132.200242] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2139.720853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.132@tcp [ 2149.024536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2149.102311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2149.152294] LNet: Added LNI 192.168.203.32@tcp1 [8/256/0/180] [ 2150.167670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2150.921697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.132@tcp [ 2151.171586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2151.208676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2151.243981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2151.279489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2151.325455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2151.368099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2151.412679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2151.442270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2151.461994] LNet: 124559:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.488483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2151.533850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2151.585092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2151.620761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2151.650760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2151.680636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2152.339496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2152.645151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2152.703398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2152.736543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2152.740061] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.132@tcp, 192.168.203.132@tcp1 [ 2152.777336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2152.839601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2152.880613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2152.916991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2152.945185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2152.961354] LNet: 124677:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2153.000723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2153.037993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2153.070478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2153.107511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2153.147863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2153.186630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2153.918797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.132@tcp [ 2154.183558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2154.218713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2154.253358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2154.285917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2154.320877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2154.367570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2154.403426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2154.438682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2154.452848] LNet: 124794:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2154.480429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2154.507363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2154.547133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2154.570791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2154.593727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2154.621100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2155.191375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2155.437390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2155.488243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2155.537507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2155.573789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2155.647669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2155.697817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2155.760579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2155.822995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2155.856816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2155.900665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2155.922930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2155.953963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2155.993400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2156.025789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2161.655194] LNet: 124996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2161.672506] LNetError: 124996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2161.693637] LNet: Removed LNI 192.168.203.32@tcp [ 2161.713922] LNet: Removed LNI 192.168.203.32@tcp1 [ 2162.064897] Key type .llcrypt unregistered [ 2162.067895] Key type ._llcrypt unregistered [ 2179.870061] Key type ._llcrypt registered [ 2179.874255] Key type .llcrypt registered [ 2181.364295] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 09:23:58 (1789133038) [ 2181.811387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2181.826713] alg: No test for adler32 (adler32-zlib) [ 2182.791647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.876715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.913772] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2183.774882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2183.809904] LNet: Added LNI 192.168.203.32@tcp2 [8/256/0/180] [ 2184.648563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.32@tcp [ 2184.737748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2185.823189] LNet: Removed LNI 192.168.203.32@tcp [ 2192.183114] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 09:24:09 (1789133049) [ 2193.569123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2195.403783] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:24:12 (1789133052) [ 2202.049961] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:24:19 (1789133059) [ 2209.289895] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:24:26 (1789133066) [ 2216.279071] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:24:33 (1789133073) [ 2223.629337] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:24:40 (1789133080) [ 2230.740082] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:24:47 (1789133087) [ 2237.581742] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:24:54 (1789133094) [ 2244.040299] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:25:01 (1789133101) [ 2250.652699] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:25:07 (1789133107) [ 2257.125889] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:25:14 (1789133114) [ 2263.440631] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:25:20 (1789133120) [ 2269.898499] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:25:26 (1789133126) [ 2275.960434] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:25:33 (1789133133) [ 2281.785135] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:25:38 (1789133138) [ 2288.337097] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:25:45 (1789133145) [ 2295.719216] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:25:52 (1789133152) [ 2303.200315] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:26:00 (1789133160) [ 2310.629451] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:26:07 (1789133167) [ 2310.768329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.800058] LNet: 136681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.814713] LNetError: 136681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.844918] LNet: Removed LNI 192.168.203.32@tcp2 [ 2310.909889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.018546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2311.040315] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2312.284631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.314228] LNet: 136828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.321536] LNetError: 136828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.333687] LNet: Removed LNI 192.168.203.32@tcp [ 2312.367494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.434963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2312.452676] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2319.244568] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:26:16 (1789133176) [ 2325.736269] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:26:22 (1789133182) [ 2332.470942] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:26:29 (1789133189) [ 2339.237097] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:26:36 (1789133196) [ 2345.938807] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:26:42 (1789133202) [ 2353.171544] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:26:50 (1789133210) [ 2360.284588] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:26:57 (1789133217) [ 2367.255185] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:27:04 (1789133224) [ 2373.425688] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:27:10 (1789133230) [ 2379.746784] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:27:16 (1789133236) [ 2385.565349] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:27:22 (1789133242) [ 2391.593679] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:27:28 (1789133248) [ 2397.176202] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2158 sec ========== 09:27:34 (1789133254) [ 2397.480962] LNet: 144291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2397.490485] LNetError: 144291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2397.499771] LNet: Removed LNI 192.168.203.32@tcp [ 2398.153558] Key type .llcrypt unregistered [ 2398.156708] Key type ._llcrypt unregistered [ 2409.471139] Key type ._llcrypt registered [ 2409.473709] Key type .llcrypt registered [ 2409.694885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2409.718033] alg: No test for adler32 (adler32-zlib) [ 2410.805693] Lustre: Lustre: Build Version: 2.17.57_132_gd76390e [ 2411.090282] LNet: Added LNI 192.168.203.32@tcp [8/256/0/180] [ 2412.775981] Key type lgssc registered [ 2413.955892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2474.875095] Lustre: Mounted lustre-client - version 2.17.57_132_gd76390e [ 2479.750365] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2485.903451] Lustre: Unmounted lustre-client [ 2513.766393] Key type lgssc unregistered [ 2514.010080] LNet: 147249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2514.019732] LNetError: 147249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2515.047210] LNet: Removed LNI 192.168.203.32@tcp [ 2515.756165] Key type .llcrypt unregistered [ 2515.759925] Key type ._llcrypt unregistered