[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 425134396 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003100] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007754] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010140] LSM: Security Framework initializing [ 0.011053] Yama: becoming mindful. [ 0.012039] SELinux: Initializing. [ 0.014061] *** VALIDATE selinux *** [ 0.022606] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026885] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027184] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030173] *** VALIDATE tmpfs *** [ 0.032604] *** VALIDATE proc *** [ 0.033376] *** VALIDATE cgroup *** [ 0.034014] *** VALIDATE cgroup2 *** [ 0.035277] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039026] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043268] debug: unmapping init [mem 0xffffffff99e59000-0xffffffff99e60fff] [ 0.046184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.048942] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051011] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054304] rcu: Hierarchical SRCU implementation. [ 0.056397] smp: Bringing up secondary CPUs ... [ 0.057605] x86: Booting SMP configuration: [ 0.058025] .... node #0, CPUs: #1 #2 #3 [ 0.061696] smp: Brought up 1 node, 4 CPUs [ 0.062845] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152500] node 0 deferred pages initialised in 86ms [ 0.155208] devtmpfs: initialized [ 0.157301] x86/mm: Memory block size: 128MB [ 0.160948] gcov: version magic: 0x41383552 [ 0.164135] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.168112] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.170294] pinctrl core: initialized pinctrl subsystem [ 0.172197] [ 0.172865] ************************************************************* [ 0.176014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.178012] ** ** [ 0.180010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.182013] ** ** [ 0.185013] ** This means that this kernel is built to expose internal ** [ 0.187010] ** IOMMU data structures, which may compromise security on ** [ 0.189012] ** your system. ** [ 0.192014] ** ** [ 0.194010] ** If you see this message and you are not debugging the ** [ 0.196013] ** kernel, report this immediately to your vendor! ** [ 0.198012] ** ** [ 0.201012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.204013] ************************************************************* [ 0.206699] NET: Registered protocol family 16 [ 0.208425] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218129] cpuidle: using governor menu [ 0.219688] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.220462] PCI: Using configuration type 1 for base access [ 0.221128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.230125] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.233013] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.236187] cryptd: max_cpu_qlen set to 1000 [ 0.239376] ACPI: Added _OSI(Module Device) [ 0.242019] ACPI: Added _OSI(Processor Device) [ 0.243013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.245012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.250365] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256386] ACPI: Interpreter enabled [ 0.258065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260012] ACPI: Using IOAPIC for interrupt routing [ 0.262098] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265299] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.273744] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.279017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.283089] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.287266] acpiphp: Slot [2] registered [ 0.289103] acpiphp: Slot [5] registered [ 0.290094] acpiphp: Slot [6] registered [ 0.291082] acpiphp: Slot [3] registered [ 0.293090] acpiphp: Slot [4] registered [ 0.294088] acpiphp: Slot [7] registered [ 0.296111] acpiphp: Slot [8] registered [ 0.298132] acpiphp: Slot [9] registered [ 0.299368] acpiphp: Slot [10] registered [ 0.301113] acpiphp: Slot [11] registered [ 0.303123] acpiphp: Slot [12] registered [ 0.304115] acpiphp: Slot [13] registered [ 0.306168] acpiphp: Slot [14] registered [ 0.308094] acpiphp: Slot [15] registered [ 0.309084] acpiphp: Slot [16] registered [ 0.311095] acpiphp: Slot [17] registered [ 0.312072] acpiphp: Slot [18] registered [ 0.314126] acpiphp: Slot [19] registered [ 0.315092] acpiphp: Slot [20] registered [ 0.317125] acpiphp: Slot [21] registered [ 0.318064] acpiphp: Slot [22] registered [ 0.319083] acpiphp: Slot [23] registered [ 0.321107] acpiphp: Slot [24] registered [ 0.322094] acpiphp: Slot [25] registered [ 0.323078] acpiphp: Slot [26] registered [ 0.324075] acpiphp: Slot [27] registered [ 0.326113] acpiphp: Slot [28] registered [ 0.327094] acpiphp: Slot [29] registered [ 0.329112] acpiphp: Slot [30] registered [ 0.330116] acpiphp: Slot [31] registered [ 0.332064] PCI host bridge to bus 0000:00 [ 0.333017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349933] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353380] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362014] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.367013] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.373017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375647] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.379767] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383076] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.385923] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.391021] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.403025] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.407571] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.414782] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.444016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.454276] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.469022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.484024] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495168] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.497343] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.500334] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.503409] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.505358] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.512127] iommu: Default domain type: Passthrough [ 0.514654] SCSI subsystem initialized [ 0.516236] ACPI: bus type USB registered [ 0.518145] usbcore: registered new interface driver usbfs [ 0.520155] usbcore: registered new interface driver hub [ 0.522140] usbcore: registered new device driver usb [ 0.525193] pps_core: LinuxPPS API ver. 1 registered [ 0.527017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.529059] PTP clock support registered [ 0.531216] EDAC MC: Ver: 3.0.0 [ 0.533138] PCI: Using ACPI for IRQ routing [ 0.534857] NetLabel: Initializing [ 0.537014] NetLabel: domain hash size = 128 [ 0.539010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.541141] NetLabel: unlabeled traffic allowed by default [ 0.544072] vgaarb: loaded [ 0.545262] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.547013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.555401] clocksource: Switched to clocksource kvm-clock [ 0.658378] VFS: Disk quotas dquot_6.6.0 [ 0.659673] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.661730] *** VALIDATE ramfs *** [ 0.662715] *** VALIDATE hugetlbfs *** [ 0.663959] pnp: PnP ACPI init [ 0.665907] pnp: PnP ACPI: found 6 devices [ 0.681419] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.685258] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.687881] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.690471] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.693378] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.696120] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.699346] NET: Registered protocol family 2 [ 0.701885] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.706881] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.710227] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.715486] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.718711] TCP: Hash tables configured (established 65536 bind 65536) [ 0.721431] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.724363] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.726970] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.729721] NET: Registered protocol family 1 [ 0.731989] RPC: Registered named UNIX socket transport module. [ 0.733800] RPC: Registered udp transport module. [ 0.735052] RPC: Registered tcp transport module. [ 0.736727] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.739345] NET: Registered protocol family 44 [ 0.741111] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.743477] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.747109] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.749397] PCI: CLS 0 bytes, default 64 [ 0.751152] Unpacking initramfs... [ 3.935517] debug: unmapping init [mem 0xffff9f72fcc64000-0xffff9f72fffcffff] [ 3.947539] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.952878] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.960795] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 5.936400] Initialise system trusted keyrings [ 5.942335] Key type blacklist registered [ 5.947611] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.985766] zbud: loaded [ 5.996926] *** VALIDATE nfs *** [ 5.999903] *** VALIDATE nfs4 *** [ 6.003626] pstore: using deflate compression [ 6.033200] Platform Keyring initialized [ 6.278148] NET: Registered protocol family 38 [ 6.280898] Key type asymmetric registered [ 6.285286] Asymmetric key parser 'x509' registered [ 6.288551] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.300963] io scheduler mq-deadline registered [ 6.304065] io scheduler kyber registered [ 6.306297] io scheduler bfq registered [ 6.309535] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.313373] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.317237] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.321426] ACPI: Power Button [PWRF] [ 6.327400] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.350724] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.368953] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.401922] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.441407] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.466799] Non-volatile memory driver v1.3 [ 6.480040] Linux agpgart interface v0.103 [ 6.570718] virtio_blk virtio1: [vda] 139528 512-byte logical blocks (71.4 MB/68.1 MiB) [ 6.585260] vda: detected capacity change from 0 to 71438336 [ 6.645899] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.654858] vdb: detected capacity change from 0 to 1073741824 [ 6.671491] libphy: Fixed MDIO Bus: probed [ 6.685910] usbcore: registered new interface driver usbserial_generic [ 6.693854] usbserial: USB Serial support registered for generic [ 6.702483] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.723730] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.735882] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.743830] mousedev: PS/2 mouse device common for all mice [ 6.749204] rtc_cmos 00:05: RTC can wake from S4 [ 6.756402] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.760316] rtc_cmos 00:05: registered as rtc0 [ 6.760412] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.760459] intel_pstate: CPU model not supported [ 6.771037] hid: raw HID events driver (C) Jiri Kosina [ 6.789901] usbcore: registered new interface driver usbhid [ 6.790627] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.795562] usbhid: USB HID core driver [ 6.802713] drop_monitor: Initializing network drop monitor service [ 6.807398] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.814590] Initializing XFRM netlink socket [ 6.819271] NET: Registered protocol family 10 [ 6.832774] Segment Routing with IPv6 [ 6.835307] NET: Registered protocol family 17 [ 6.849337] mpls_gso: MPLS GSO support [ 6.872695] RAS: Correctable Errors collector initialized. [ 6.880650] AVX version of gcm_enc/dec engaged. [ 6.884089] AES CTR mode by8 optimization enabled [ 7.095221] sched_clock: Marking stable (7095171605, 0)->(7998159713, -902988108) [ 7.106358] registered taskstats version 1 [ 7.108934] Loading compiled-in X.509 certificates [ 7.111335] zswap: loaded using pool lzo/zbud [ 7.151858] Key type big_key registered [ 7.172583] Key type encrypted registered [ 7.176550] ima: No TPM chip found, activating TPM-bypass! [ 7.181458] ima: Allocated hash algorithm: sha1 [ 7.187700] ima: No architecture policies found [ 7.191076] evm: Initialising EVM extended attributes: [ 7.192844] evm: security.selinux [ 7.194534] evm: security.ima [ 7.197286] evm: security.capability [ 7.199475] evm: HMAC attrs: 0x1 [ 7.204296] rtc_cmos 00:05: setting system clock to 2026-06-09 08:28:55 UTC (1780993735) [ 7.212764] debug: unmapping init [mem 0xffffffff9ae03000-0xffffffff9affffff] [ 7.221398] debug: unmapping init [mem 0xffffffff99b82000-0xffffffff99e58fff] [ 7.239503] Write protecting the kernel read-only data: 28672k [ 7.249931] debug: unmapping init [mem 0xffffffff98203000-0xffffffff983fffff] [ 7.255888] debug: unmapping init [mem 0xffffffff98b14000-0xffffffff98bfffff] [ 7.346425] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.365313] systemd[1]: Detected virtualization kvm. [ 7.369308] systemd[1]: Detected architecture x86-64. [ 7.372872] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.431375] systemd[1]: No hostname configured. [ 7.437490] systemd[1]: Set hostname to . [ 7.444452] random: systemd: uninitialized urandom read (16 bytes read) [ 7.449270] systemd[1]: Initializing machine ID from random generator. [ 7.593858] random: ln: uninitialized urandom read (6 bytes read) [ 7.787730] random: systemd: uninitialized urandom read (16 bytes read) [ 7.792790] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 7.801629] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.831194] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.275484] device-mapper: uevent: version 1.0.3 [ 9.281991] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 10.630371] virtio_net virtio0 ens2: renamed from eth0 [ 11.323670] scsi host0: ata_piix [ 11.361329] scsi host1: ata_piix [ 11.369840] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.377591] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 15.711562] random: crng init done [ 15.714201] random: 7 urandom warning(s) missed due to ratelimiting [ 16.889872] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 19.041162] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.842835] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.779829] SELinux: Disabled at runtime. [ 22.895469] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.916987] systemd[1]: Detected virtualization kvm. [ 22.921787] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 24.582774] systemd[1]: initrd-switch-root.service: Succeeded. [ 24.588816] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 24.614555] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 24.626169] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 24.636753] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 24.654702] systemd[1]: Starting Journal Service... Starting Journal Service... [ 24.666935] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting POSIX Message Queue File System... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ 25.285789] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 26.033198] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.796944] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.923305] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 27.234329] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.309413] EDAC sbridge: Ver: 1.1.2 [ 30.957638] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.832240] NFS: Registering the id_resolver key type [ 31.835746] Key type id_resolver registered [ 31.837692] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg313-client login: [ 84.896704] libcfs: loading out-of-tree module taints kernel. [ 85.110047] Key type ._llcrypt registered [ 85.112319] Key type .llcrypt registered [ 85.596337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 85.615523] alg: No test for adler32 (adler32-zlib) [ 86.881329] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 87.527627] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 89.351128] Key type lgssc registered [ 91.211460] Lustre: Echo OBD driver; http://www.lustre.org/ [ 219.751664] Lustre: Mounted lustre-client [ 225.148159] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.965500] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing check_logdir /tmp/testlogs/ [ 245.219939] Lustre: lustre-OST0000-osc-ffff9f7344e07000: disconnect after 23s idle [ 246.387783] Lustre: DEBUG MARKER: oleg313-client.virtnet: executing yml_node [ 251.181580] Lustre: DEBUG MARKER: Client: 2.17.54.1 [ 253.309696] Lustre: DEBUG MARKER: MDS: 2.17.54.1 [ 256.141432] Lustre: DEBUG MARKER: OSS: 2.17.54.1 [ 257.731036] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 9 04:33:04 EDT 2026 [ 271.661204] Lustre: Unmounted lustre-client [ 300.593812] Key type lgssc unregistered [ 300.822955] LNet: 8314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.832197] LNetError: 8314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 300.867405] LNet: Removed LNI 192.168.203.13@tcp [ 301.524914] Key type .llcrypt unregistered [ 301.532102] Key type ._llcrypt unregistered [ 326.254583] Key type ._llcrypt registered [ 326.258176] Key type .llcrypt registered [ 326.474883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 326.485948] alg: No test for adler32 (adler32-zlib) [ 327.366170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 327.411890] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 327.429124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 331.114894] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 331.281989] LNet: 9198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.297657] LNetError: 9198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 331.317938] LNet: Removed LNI 192.168.203.13@tcp [ 331.843726] Key type .llcrypt unregistered [ 331.846454] Key type ._llcrypt unregistered [ 336.659898] Key type ._llcrypt registered [ 336.663832] Key type .llcrypt registered [ 337.626377] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 04:34:24 (1780994064) [ 337.898579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 337.910239] alg: No test for adler32 (adler32-zlib) [ 338.811720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.907290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.925805] LNet: 9608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.938944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 339.002461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 339.021142] LNet: 9615:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 344.747791] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 04:34:31 (1780994071) [ 344.886666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.903906] LNet: 10194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.933504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.042090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 350.762573] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 04:34:37 (1780994077) [ 350.994661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.010460] LNet: 10781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 351.027099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.087423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 356.830022] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 04:34:43 (1780994083) [ 356.982255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.015128] LNet: 11364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.078216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.214667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 362.128500] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 04:34:49 (1780994089) [ 362.312887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.357866] LNet: 11950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.389659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.498463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 362.874445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 368.656366] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 04:34:55 (1780994095) [ 368.753743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.803865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.903637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 369.325796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 369.699710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 376.594520] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 04:35:03 (1780994103) [ 376.698699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.733785] LNet: 13132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.739343] LNet: 13132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 376.772553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.878158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 382.964805] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 04:35:09 (1780994109) [ 383.060545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.134809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.259650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 383.308585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 383.724810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 383.754418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 384.204497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 384.251217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 384.689185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 384.724128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 385.187594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 385.247967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 385.683851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 385.725717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.163122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 386.223343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 386.809283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 386.858310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.268274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 387.300548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 387.616584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 387.673383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.191986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 388.273342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 388.675688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 388.706610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 389.168907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 389.215367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 395.509656] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 04:35:22 (1780994122) [ 395.617550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.662256] LNet: 14397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.670737] LNet: 14397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 395.729272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.894791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 395.949342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 402.652790] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 04:35:29 (1780994129) [ 402.739257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.787102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.845914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 402.869881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 409.029487] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 04:35:36 (1780994136) [ 409.167585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 409.280759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 409.412387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 409.444097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 415.997306] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 04:35:43 (1780994143) [ 416.343649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.389895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.480296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 416.530263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 422.403940] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 04:35:49 (1780994149) [ 422.490229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.537255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.615109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 422.667965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 429.272654] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 04:35:56 (1780994156) [ 429.382632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.411051] LNet: 17328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 429.418308] LNet: 17328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 429.457839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 429.559989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 429.597204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 435.861083] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 04:36:02 (1780994162) [ 435.976567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.043754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.140388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.719728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 444.234767] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 04:36:10 (1780994170) [ 444.417063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.454916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.687885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.965989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 451.651721] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 04:36:18 (1780994178) [ 451.835783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.901596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.102201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.330736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.138644] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 04:36:25 (1780994185) [ 458.363940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.430937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.049378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 459.329506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 464.910414] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 04:36:32 (1780994192) [ 465.185601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.238756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.310794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 465.353841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.810088] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 04:36:38 (1780994198) [ 470.900755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.955097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.047652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.095493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.399207] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 04:36:43 (1780994203) [ 476.559594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.599636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.678477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 481.685477] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 04:36:48 (1780994208) [ 481.847706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.885135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.973986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 487.253872] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 04:36:54 (1780994214) [ 487.421986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.466685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.527972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 487.747571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 492.504827] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 04:36:59 (1780994219) [ 492.636735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.696840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.761739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 492.800280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 497.821225] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 04:37:05 (1780994225) [ 497.971258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.994659] LNet: 23944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.999711] LNet: 23944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 498.017742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.065524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 498.086360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 503.188740] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 04:37:10 (1780994230) [ 503.523067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.595667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.715599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 503.772980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 509.586345] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 04:37:16 (1780994236) [ 509.653916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.723198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.816879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 509.872571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 516.330345] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 04:37:23 (1780994243) [ 516.516759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.591226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.670150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 516.760970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 516.790318] LNetError: 25717:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 516.834474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 516.866736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 523.185366] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 04:37:30 (1780994250) [ 523.439205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.467750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.569506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.O0oZHv03lp [ 523.694445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.O0oZHv03lp [ 523.749914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.O0oZHv03lp [ 530.376955] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 04:37:37 (1780994257) [ 530.665164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.705805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.771242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 530.816667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 530.863246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 538.163638] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 04:37:45 (1780994265) [ 538.258813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.292556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.381585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 540.702919] LNet: Message forwarding will be enabled in 2s [ 543.775118] LNet: Message forwarding enabled [ 544.196335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 544.225605] LNet: Message forwarding will stop in 2s to 6s [ 548.680347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 549.228987] LNet: Message forwarding disabled [ 550.715543] LNet: Message forwarding will be enabled in 2s [ 554.339113] LNet: Message forwarding enabled [ 557.127095] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 04:38:04 (1780994284) [ 557.297396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.628299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.726579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 564.009571] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 04:38:11 (1780994291) [ 564.130168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.164203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.271401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 564.304067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 564.347868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 564.379492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 564.418925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 564.457951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 564.488312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 564.538220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 564.573770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 564.613547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 564.663165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 564.685808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 564.734811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 564.793885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 564.842110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 564.883931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 564.930612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 564.988262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 565.029679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 565.067617] LNetError: 28786:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 570.458193] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 04:38:17 (1780994297) [ 570.615198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.669120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.724502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 576.219619] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 04:38:23 (1780994303) [ 577.935493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg313-server [ 579.650839] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 04:38:26 (1780994306) [ 579.833114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.886417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.937542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 579.967645] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 581.726113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[14-18]@tcp [ 581.763917] LNet: 30502:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 581.836974] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.14@tcp is being used as a gateway but routing feature is not turned on [ 581.854973] LNetError: 30292:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.14@tcp has gone from up to down [ 582.819641] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.14@tcp is being used as a gateway but routing feature is not turned on [ 582.845068] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 583.845676] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 583.861118] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 585.896916] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.14@tcp is being used as a gateway but routing feature is not turned on [ 585.910234] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 587.429477] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 04:38:34 (1780994314) [ 588.744388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg313-server [ 589.992137] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 590.002758] LNetError: 30292:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 590.114164] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 04:38:37 (1780994317) [ 590.279381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.302933] LNetError: 31437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 590.328567] LNet: Removed LNI 192.168.203.13@tcp [ 590.356697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.403700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 590.431789] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 591.022422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[14-19/2]@tcp [ 591.053339] LNet: 31554:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 591.063614] LNet: 31554:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 591.086779] LNetError: 31441:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.18@tcp has gone from up to down [ 591.089131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[14-19/2]@tcp [ 591.103402] LNetError: 31441:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 596.145775] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 04:38:43 (1780994323) [ 596.343146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.363131] LNetError: 32146:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 596.383521] LNet: Removed LNI 192.168.203.13@tcp [ 596.414734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.510599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 596.585988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.642664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.709279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 596.772605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.824505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.885864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 596.972374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.024923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.089759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 597.151697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.197169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.244095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 597.325644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.354718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.396271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 597.465900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.495885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.542213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 597.617282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.669085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.780399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 597.868986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.895938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.936981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 598.002698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.033553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.100220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 598.186511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.231865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.293421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 598.350374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 598.382634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 598.434487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.921164] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 04:38:50 (1780994330) [ 605.728060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg313-server [ 607.210980] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 04:38:54 (1780994334) [ 609.151666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg313-server [ 610.945450] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 04:38:57 (1780994337) [ 612.138977] Key type .llcrypt unregistered [ 612.141399] Key type ._llcrypt unregistered [ 619.058754] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 619.376845] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 619.392334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 619.402290] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 622.303874] Key type ._llcrypt registered [ 622.306763] Key type .llcrypt registered [ 622.545962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 622.566722] alg: No test for adler32 (adler32-zlib) [ 623.488754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.572581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.620198] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 624.505379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 624.530382] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 626.919319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 626.935725] LNet: Removed LNI 10.1.2.3@tcp [ 627.097699] LNet: 34268:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.108709] LNetError: 34268:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 627.126197] LNet: Removed LNI 192.168.203.13@tcp [ 628.025796] Key type .llcrypt unregistered [ 628.027312] Key type ._llcrypt unregistered [ 641.330586] Key type ._llcrypt registered [ 641.333279] Key type .llcrypt registered [ 642.792315] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 04:39:29 (1780994369) [ 643.328712] Key type .llcrypt unregistered [ 643.333625] Key type ._llcrypt unregistered [ 648.886605] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 649.028481] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 649.031674] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 649.046026] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 651.654837] Key type ._llcrypt registered [ 651.659618] Key type .llcrypt registered [ 651.875940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 651.898760] alg: No test for adler32 (adler32-zlib) [ 652.825518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.895167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 652.910962] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 653.406391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 653.425145] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 656.721887] LNet: 35758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.726293] LNetError: 35758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.732046] LNet: Removed LNI 192.168.203.13@tcp [ 656.746531] LNet: Removed LNI 10.1.2.3@tcp [ 657.343517] Key type .llcrypt unregistered [ 657.348380] Key type ._llcrypt unregistered [ 668.106224] Key type ._llcrypt registered [ 668.110926] Key type .llcrypt registered [ 669.607663] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 04:39:56 (1780994396) [ 670.049619] Key type .llcrypt unregistered [ 670.054339] Key type ._llcrypt unregistered [ 676.050877] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 676.175798] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 676.182913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 676.192167] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 678.833728] Key type ._llcrypt registered [ 678.835890] Key type .llcrypt registered [ 678.977576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 678.996587] alg: No test for adler32 (adler32-zlib) [ 679.911866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.963726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 679.986256] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 681.814212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 681.842046] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 683.966534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 684.004251] LNet: Removed LNI 10.1.2.3@tcp [ 684.022252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 684.071729] LNet: Removed LNI 10.1.2.31@tcp [ 684.098636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 684.133295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.896081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 684.916047] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 685.770454] LNet: 37681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.779716] LNetError: 37681:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 685.786202] LNet: Removed LNI 10.1.2.3@tcp [ 685.801795] LNet: Removed LNI 10.1.2.31@tcp [ 686.390550] Key type .llcrypt unregistered [ 686.396924] Key type ._llcrypt unregistered [ 697.116875] Key type ._llcrypt registered [ 697.119759] Key type .llcrypt registered [ 698.297906] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 04:40:25 (1780994425) [ 698.679080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 698.697430] alg: No test for adler32 (adler32-zlib) [ 699.640416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.688966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.714110] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 700.450373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 700.476772] LNet: 38635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.486848] LNetError: 38635:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.513634] LNet: Removed LNI 192.168.203.13@tcp [ 700.530091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.579969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.13@tcp [ 700.609580] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 701.331585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.13@tcp [ 701.374810] LNet: Removed LNI 192.168.203.13@tcp [ 701.465160] LNet: 38770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.119140] Key type .llcrypt unregistered [ 702.124195] Key type ._llcrypt unregistered [ 712.833652] Key type ._llcrypt registered [ 712.836191] Key type .llcrypt registered [ 714.121942] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 04:40:40 (1780994440) [ 715.797490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg313-server [ 717.200642] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 04:40:44 (1780994444) [ 717.672449] Key type .llcrypt unregistered [ 717.676318] Key type ._llcrypt unregistered [ 723.352459] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 723.535190] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 723.540107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 723.547889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.284883] Key type ._llcrypt registered [ 726.290985] Key type .llcrypt registered [ 726.577310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.604650] alg: No test for adler32 (adler32-zlib) [ 727.506306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.560233] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 728.035810] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 729.878853] LNet: 40600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.885887] LNetError: 40600:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.897138] LNet: Removed LNI 192.168.203.13@tcp1 [ 729.919270] LNet: Removed LNI 10.1.2.3@tcp2 [ 730.461472] Key type .llcrypt unregistered [ 730.463802] Key type ._llcrypt unregistered [ 739.259455] Key type ._llcrypt registered [ 739.263168] Key type .llcrypt registered [ 740.351811] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 04:41:07 (1780994467) [ 740.758482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 740.783378] alg: No test for adler32 (adler32-zlib) [ 741.665753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.705697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 741.729103] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 742.495820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.514294] LNet: 41570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.520700] LNetError: 41570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.529784] LNet: Removed LNI 192.168.203.13@tcp [ 742.552976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.589872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 742.620516] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 748.204674] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 04:41:15 (1780994475) [ 748.368924] LNet: 42159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.382900] LNetError: 42159:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.420232] LNet: Removed LNI 192.168.203.13@tcp [ 749.035470] Key type .llcrypt unregistered [ 749.039658] Key type ._llcrypt unregistered [ 753.795950] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 753.874454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 753.883275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 753.895948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.361551] Key type ._llcrypt registered [ 756.364350] Key type .llcrypt registered [ 756.491351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.504722] alg: No test for adler32 (adler32-zlib) [ 757.392511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.456624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 757.480468] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 758.018291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.033796] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.667528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 759.694753] LNet: 42904:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.702684] LNetError: 42904:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.707454] LNet: Removed LNI 192.168.203.13@tcp [ 759.719407] LNet: Removed LNI 10.1.2.3@tcp [ 759.753034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.812198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 759.829406] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 759.834498] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.178708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 760.196450] LNet: 42931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 760.207300] LNetError: 42931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 760.215147] LNetError: 42931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 760.220068] LNet: Removed LNI 192.168.203.13@tcp [ 760.229622] LNet: Removed LNI 10.1.2.3@tcp [ 760.278580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.348967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 760.382129] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 760.386153] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.762115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 760.799695] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 761.553362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 761.574257] LNet: 43093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.577712] LNetError: 43093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 761.581626] LNetError: 43093:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 761.586696] LNet: Removed LNI 192.168.203.13@tcp [ 761.588585] LNet: Removed LNI 10.1.2.3@tcp [ 761.601536] LNet: Removed LNI 192.168.203.13@tcp2 [ 761.630239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 761.698957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 761.727312] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 761.740947] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.751556] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 767.199320] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 04:41:34 (1780994494) [ 767.280467] LNet: 43682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.284797] LNetError: 43682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.289755] LNetError: 43682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 767.295787] LNet: Removed LNI 192.168.203.13@tcp [ 767.297976] LNet: Removed LNI 10.1.2.3@tcp [ 767.304100] LNet: Removed LNI 192.168.203.13@tcp2 [ 767.829726] Key type .llcrypt unregistered [ 767.833426] Key type ._llcrypt unregistered [ 772.841376] Key type ._llcrypt registered [ 772.843995] Key type .llcrypt registered [ 772.996842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.014442] alg: No test for adler32 (adler32-zlib) [ 773.888237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.927757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 773.951280] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 774.677843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.697900] LNet: 44085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.702993] LNetError: 44085:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 774.720320] LNet: Removed LNI 192.168.203.13@tcp [ 774.737937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.772456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 774.795085] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 775.115896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.139438] LNet: Removed LNI 192.168.203.13@tcp [ 775.155512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.189115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 775.223117] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 775.669934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.700184] LNet: 44145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.708159] LNet: 44145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 775.713145] LNetError: 44145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.722542] LNetError: 44145:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 775.739068] LNet: Removed LNI 192.168.203.13@tcp [ 775.765148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.812375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 775.835717] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 776.248562] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.302070] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.304343] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.310457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.784323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 778.799993] LNet: 44222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.805978] LNetError: 44222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.820434] LNet: Removed LNI 192.168.203.13@tcp [ 778.840376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.923689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 778.956354] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 787.493416] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 04:41:54 (1780994514) [ 787.577111] LNet: 44811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.581375] LNetError: 44811:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.616097] LNet: Removed LNI 192.168.203.13@tcp [ 788.256548] Key type .llcrypt unregistered [ 788.262994] Key type ._llcrypt unregistered [ 795.086153] Key type ._llcrypt registered [ 795.089229] Key type .llcrypt registered [ 795.329519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 795.340398] alg: No test for adler32 (adler32-zlib) [ 796.254649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.320969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.345940] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 797.548054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 797.571971] LNet: 45218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.587701] LNetError: 45218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 797.616040] LNet: Removed LNI 192.168.203.13@tcp [ 797.643536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.755758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 797.787485] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 805.488630] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 04:42:12 (1780994532) [ 805.588515] LNet: 45802:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.598444] LNetError: 45802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 805.613565] LNet: Removed LNI 192.168.203.13@tcp [ 806.298015] Key type .llcrypt unregistered [ 806.301067] Key type ._llcrypt unregistered [ 813.707560] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 813.891869] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 813.905054] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 813.931495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.676466] Key type ._llcrypt registered [ 816.680968] Key type .llcrypt registered [ 816.984359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.016843] alg: No test for adler32 (adler32-zlib) [ 818.097635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.171245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.197169] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 819.187651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.207309] LNet: 46262:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.213895] LNetError: 46262:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.225363] LNet: Removed LNI 192.168.203.13@tcp [ 819.255548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.319837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 819.344131] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 826.387619] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 04:42:32 (1780994552) [ 826.563398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.589960] LNet: 46860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.618076] LNetError: 46860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.647092] LNet: Removed LNI 192.168.203.13@tcp [ 826.684305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.788254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 833.948790] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 04:42:40 (1780994560) [ 834.096704] LNet: 47440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.706602] Key type .llcrypt unregistered [ 834.714446] Key type ._llcrypt unregistered [ 840.483316] Key type ._llcrypt registered [ 840.488560] Key type .llcrypt registered [ 840.784746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 840.800951] alg: No test for adler32 (adler32-zlib) [ 841.649766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 841.680810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 841.698907] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 842.328857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.363943] LNet: 47840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.377446] LNetError: 47840:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.394647] LNet: Removed LNI 192.168.203.13@tcp [ 842.427653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.504935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 842.537221] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 848.625815] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 04:42:55 (1780994575) [ 848.917459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.935573] LNet: 48433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.946599] LNetError: 48433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.965742] LNet: Removed LNI 192.168.203.13@tcp [ 848.995689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.088847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 855.314405] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 04:43:02 (1780994582) [ 856.704760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 858.715730] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 04:43:05 (1780994585) [ 859.960759] hrtimer: interrupt took 4617080 ns [ 860.116993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 862.007292] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 04:43:08 (1780994588) [ 863.518581] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 865.227831] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 04:43:12 (1780994592) [ 866.946464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 868.724522] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 04:43:15 (1780994595) [ 870.526869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 872.259930] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 04:43:19 (1780994599) [ 872.473787] LNet: 50482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.084625] Key type .llcrypt unregistered [ 873.092507] Key type ._llcrypt unregistered [ 878.562327] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 878.719337] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 878.727642] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 878.733574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.247174] Key type ._llcrypt registered [ 881.249281] Key type .llcrypt registered [ 881.381742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.402242] alg: No test for adler32 (adler32-zlib) [ 882.281558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.331374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.347575] LNet: Added LNI 192.168.203.13@tcp [8/512/0/180] [ 882.352351] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 882.356693] LNet: 50804:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 882.448898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.464241] LNet: 50823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.468803] LNetError: 50823:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.477885] LNet: Removed LNI 192.168.203.13@tcp [ 882.483820] LNetError: 50823:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.496900] LNet: Removed LNI 10.1.2.3@tcp [ 882.531482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.586078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.612918] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 882.663615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.685945] LNet: 50847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.691243] LNetError: 50847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.701810] LNet: Removed LNI 192.168.203.13@tcp [ 882.721620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.774898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 882.790736] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 882.794515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 882.797542] LNet: 50856:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 882.863923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.882841] LNet: 50874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.887925] LNetError: 50874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.891646] LNet: Removed LNI 192.168.203.13@tcp [ 882.893869] LNetError: 50874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.910637] LNet: Removed LNI 10.1.2.3@tcp [ 882.941348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.987565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 883.003581] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 883.007398] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.010400] LNet: 50882:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 883.164730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.182133] LNet: 50908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.188823] LNetError: 50908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.193620] LNet: Removed LNI 192.168.203.13@tcp [ 883.196760] LNetError: 50908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.209262] LNet: Removed LNI 10.1.2.3@tcp [ 883.233814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.305360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 883.333906] LNet: Added LNI 192.168.203.13@tcp [8/256/0/182] [ 883.349782] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 883.432735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.458331] LNet: 50936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.465971] LNetError: 50936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.469833] LNet: Removed LNI 192.168.203.13@tcp [ 883.472372] LNetError: 50936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.486703] LNet: Removed LNI 10.1.2.3@tcp [ 883.503667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.573311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 883.597255] LNet: Added LNI 192.168.203.13@tcp [8/256/0/182] [ 883.601244] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 883.608208] LNet: 50944:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 883.677630] LNet: Removed LNI 192.168.203.13@tcp [ 883.690547] LNet: Removed LNI 10.1.2.3@tcp [ 884.199147] Key type .llcrypt unregistered [ 884.201070] Key type ._llcrypt unregistered [ 894.162231] Key type ._llcrypt registered [ 894.164844] Key type .llcrypt registered [ 895.299348] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 04:43:42 (1780994622) [ 895.732805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 895.751589] alg: No test for adler32 (adler32-zlib) [ 896.647562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.721588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.753487] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 897.539454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.13 [ 897.561328] LNet: 51919:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 897.576705] LNet: Removed LNI 192.168.203.13@tcp [ 903.740655] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 04:43:50 (1780994630) [ 904.040282] LNet: 52491:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.048921] LNetError: 52491:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.060340] LNet: Removed LNI 192.168.203.13@tcp [ 904.971280] Key type .llcrypt unregistered [ 904.979901] Key type ._llcrypt unregistered [ 911.145310] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 911.517521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 914.447571] Key type ._llcrypt registered [ 914.452888] Key type .llcrypt registered [ 914.717845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.743292] alg: No test for adler32 (adler32-zlib) [ 916.207729] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 922.915301] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 04:44:09 (1780994649) [ 923.094571] LNet: 53404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.113776] LNetError: 53404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 923.138506] LNet: Removed LNI 10.1.2.3@tcp [ 923.636548] Key type .llcrypt unregistered [ 923.641161] Key type ._llcrypt unregistered [ 928.744203] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 928.967699] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 928.971734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 928.982400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.675804] Key type ._llcrypt registered [ 931.682602] Key type .llcrypt registered [ 931.839237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.848666] alg: No test for adler32 (adler32-zlib) [ 932.812992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 938.801156] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 04:44:25 (1780994665) [ 940.596288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 942.370614] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 04:44:29 (1780994669) [ 942.632406] LNet: 54589:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.647907] LNetError: 54589:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 942.682848] LNet: Removed LNI 10.1.2.3@tcp [ 943.292682] Key type .llcrypt unregistered [ 943.294519] Key type ._llcrypt unregistered [ 948.715262] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.906548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.909495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 948.912330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.662652] Key type ._llcrypt registered [ 951.669050] Key type .llcrypt registered [ 951.919209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.935323] alg: No test for adler32 (adler32-zlib) [ 952.862922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.959590] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.135145] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 04:44:47 (1780994687) [ 960.526956] LNet: 55819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.535667] LNetError: 55819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 960.552045] LNet: Removed LNI 10.1.2.3@tcp [ 961.056836] Key type .llcrypt unregistered [ 961.060768] Key type ._llcrypt unregistered [ 966.340693] Key type ._llcrypt registered [ 966.342834] Key type .llcrypt registered [ 966.515457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 966.526950] alg: No test for adler32 (adler32-zlib) [ 967.413313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 967.437950] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 974.399645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 975.105041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_interrupt -r 1 [ 975.138702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_interrupt -r 1 [ 975.169655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 975.189293] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 975.406087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 975.433359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 975.455602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 975.477318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 975.508604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 975.536614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 975.571813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 975.610844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 975.629767] LNet: 56256:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 975.647195] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 975.650422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 975.682560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 975.726816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 975.764776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 975.797936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 975.831556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 975.864890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 976.419578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_dropped -r 1 [ 976.451041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_dropped -r 1 [ 976.480329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 976.500567] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 976.679989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 976.704931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 976.734755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 976.763603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 976.793895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 976.824064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 976.848454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 976.876379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 976.892074] LNet: 56366:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 976.906944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 976.936644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 976.965967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 976.992836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 977.029985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.078270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 977.126986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 977.686368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_aborted -r 1 [ 977.744191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_aborted -r 1 [ 977.780692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 977.803222] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 977.809247] LNet: Skipped 1 previous similar message [ 978.047383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 978.099970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 978.154862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 978.210598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 978.275212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 978.337816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 978.379866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 978.421784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 978.439973] LNet: 56475:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 978.460154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 978.506864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 978.551716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 978.588578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 978.640605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 978.677467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 978.705365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 979.367375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_no_route -r 1 [ 979.438196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_no_route -r 1 [ 979.509735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 979.828680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 979.873578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 979.927401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 979.958682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 979.990866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 980.020636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 980.045232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 980.068410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 980.095760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 980.136907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 980.166897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 980.215323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 980.257106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 980.311110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 980.354088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 980.958307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_timeout -r 1 [ 981.010503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_timeout -r 1 [ 981.070863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 981.107737] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 981.122856] LNet: Skipped 1 previous similar message [ 981.453101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.494416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 981.546562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.580995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.632944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.664873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 981.692917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.720606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.741476] LNet: 56694:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.748475] LNet: 56694:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 981.767991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.814599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.855108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.895685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.931854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.981456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 982.038771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 982.620960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_error -r 1 [ 982.674603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_error -r 1 [ 982.715593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 982.905438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 982.932453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 982.973661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.004467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.033286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.060235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 983.082865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 983.112423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 983.156372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 983.185069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 983.228625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 983.273906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 983.302221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 983.369632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 983.405969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 983.665605] LNet: 56844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.672502] LNetError: 56844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.687415] LNet: Removed LNI 192.168.203.13@tcp [ 984.093918] Key type .llcrypt unregistered [ 984.096598] Key type ._llcrypt unregistered [ 999.965285] Key type ._llcrypt registered [ 999.966708] Key type .llcrypt registered [ 1001.099603] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 04:45:28 (1780994728) [ 1001.698414] Key type .llcrypt unregistered [ 1001.701372] Key type ._llcrypt unregistered [ 1007.627527] Key type ._llcrypt registered [ 1007.629352] Key type .llcrypt registered [ 1007.897612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.914994] alg: No test for adler32 (adler32-zlib) [ 1008.894077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1008.926726] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1017.396710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1031.286492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1031.354453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1031.374303] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1032.967850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_interrupt -r 1 [ 1033.041972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 1 [ 1033.099295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_interrupt -r 1 [ 1033.162838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 1 [ 1033.223960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_interrupt -r 1 [ 1033.294599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 1 [ 1033.331311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_interrupt -r 1 [ 1033.361964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 1 [ 1033.405650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1033.438570] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1033.456113] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1033.673020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.708857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.740762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.766404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.813435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.864643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.897377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.935544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.955264] LNet: 58314:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1033.978431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.021389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.057761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.078780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.105622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.133761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.174463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.706323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_dropped -r 1 [ 1034.741287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 1 [ 1034.801802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_dropped -r 1 [ 1034.849169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 1 [ 1034.918479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_dropped -r 1 [ 1034.976690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 1 [ 1035.030671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_dropped -r 1 [ 1035.069950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 1 [ 1035.105819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1035.140030] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1035.146939] LNet: Skipped 2 previous similar messages [ 1035.400259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.426897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.480757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.522967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.576400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.629944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.682586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.726508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.764766] LNet: 58435:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.791592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.827797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.856755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.910444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.971646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.011744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.058331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.572277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_aborted -r 1 [ 1036.637133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 1 [ 1036.690715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_aborted -r 1 [ 1036.752587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 1 [ 1036.815661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_aborted -r 1 [ 1036.871575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 1 [ 1036.904663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_aborted -r 1 [ 1036.942745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 1 [ 1036.997741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1037.026587] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1037.038589] LNet: Skipped 2 previous similar messages [ 1037.238834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.282713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1037.320887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.375756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.419460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.463325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.524311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.570877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1037.613052] LNet: 58557:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1037.635031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.685780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.727534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.756388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.781691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.808184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.832096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1038.329208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_no_route -r 1 [ 1038.378951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 1 [ 1038.422851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_no_route -r 1 [ 1038.468638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 1 [ 1038.512707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_no_route -r 1 [ 1038.549512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 1 [ 1038.597070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_no_route -r 1 [ 1038.635288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 1 [ 1038.710351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1039.006576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1039.045239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1039.079310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1039.128951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1039.160284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1039.203175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1039.250673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1039.281667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1039.321464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1039.354758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1039.385644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1039.412736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1039.440735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1039.472590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1039.503514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1040.076691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_timeout -r 1 [ 1040.136858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 1 [ 1040.185983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_timeout -r 1 [ 1040.238861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 1 [ 1040.287471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_timeout -r 1 [ 1040.338576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 1 [ 1040.373898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_timeout -r 1 [ 1040.411634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 1 [ 1040.446874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1040.461552] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1040.468318] LNet: Skipped 5 previous similar messages [ 1040.696662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1040.750558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1040.777302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1040.807323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1040.838151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1040.871755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1040.903582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1040.928872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1040.943384] LNet: 58799:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1040.948137] LNet: 58799:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1040.963479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1041.021246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1041.046145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1041.070837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1041.120837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1041.163478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1041.212899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1041.778112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_error -r 1 [ 1041.814564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_error -r 1 [ 1041.846581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_error -r 1 [ 1041.894887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_error -r 1 [ 1041.933933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_error -r 1 [ 1041.960882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_error -r 1 [ 1041.987426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_error -r 1 [ 1042.022742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_error -r 1 [ 1042.070079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1042.263548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1042.293134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1042.318958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1042.347595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1042.386913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1042.407283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1042.434622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1042.460894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1042.486493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1042.516513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1042.542837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1042.569336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1042.603133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1042.631437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1042.656888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1047.482381] LNet: 58996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.486499] LNetError: 58996:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.491082] LNet: Removed LNI 192.168.203.13@tcp [ 1047.506034] LNet: Removed LNI 192.168.203.13@tcp1 [ 1047.930583] Key type .llcrypt unregistered [ 1047.932987] Key type ._llcrypt unregistered [ 1064.553844] Key type ._llcrypt registered [ 1064.557540] Key type .llcrypt registered [ 1065.848634] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 04:46:32 (1780994792) [ 1066.808723] Key type .llcrypt unregistered [ 1066.812930] Key type ._llcrypt unregistered [ 1073.862784] Key type ._llcrypt registered [ 1073.864983] Key type .llcrypt registered [ 1073.998899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1074.014847] alg: No test for adler32 (adler32-zlib) [ 1074.876712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1074.906244] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1082.026446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1082.560914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_dropped -r 1 [ 1082.589561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_dropped -r 1 [ 1082.626957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1082.648397] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1082.851660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.869197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1082.885687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.924018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1082.947103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1082.975548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.000734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.029356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.063965] LNet: 60250:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.086034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.126149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.169378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.192849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.232202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.279435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.313217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1084.052431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_error -r 1 [ 1084.081981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_error -r 1 [ 1084.126899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1084.160848] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1084.403586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.427771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1084.453364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.483920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.512880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.567666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.607840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.656673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.688689] LNet: 60360:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.719728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.749493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.772988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.801487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1084.848607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1084.919787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1084.969342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1085.454120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_timeout -r 1 [ 1085.488799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_timeout -r 1 [ 1085.520734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1085.535525] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1085.846358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.902894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1085.946114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.971985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.008290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.038904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.064494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.096967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.116360] LNet: 60469:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.133128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.163206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.212900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.241576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.271809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.307671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.352319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1086.590543] LNet: 60510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.603053] LNetError: 60510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1086.625268] LNet: Removed LNI 192.168.203.13@tcp [ 1086.939493] Key type .llcrypt unregistered [ 1086.941410] Key type ._llcrypt unregistered [ 1104.079632] Key type ._llcrypt registered [ 1104.082604] Key type .llcrypt registered [ 1105.676427] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 04:47:12 (1780994832) [ 1106.660753] Key type .llcrypt unregistered [ 1106.666437] Key type ._llcrypt unregistered [ 1114.930270] Key type ._llcrypt registered [ 1114.949871] Key type .llcrypt registered [ 1115.427820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1115.475201] alg: No test for adler32 (adler32-zlib) [ 1116.514767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1116.575876] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1124.058276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1134.864689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1134.930676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1134.971068] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1137.176566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_dropped -r 1 [ 1137.229120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 1 [ 1137.290630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_dropped -r 1 [ 1137.354795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 1 [ 1137.386461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_dropped -r 1 [ 1137.441630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 1 [ 1137.498855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_dropped -r 1 [ 1137.541862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 1 [ 1137.584446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1137.612379] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1137.620168] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 1137.963720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.003533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1138.060984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.105971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1138.140407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1138.197749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1138.284634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1138.383902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1138.424718] LNet: 61978:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1138.497816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1138.540645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1138.592985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1138.624608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1138.656509] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1138.662730] LNet: Skipped 3 previous similar messages [ 1138.672643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1138.730931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1138.781390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1138.812606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.568228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_error -r 1 [ 1139.675718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_error -r 1 [ 1139.765846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_error -r 1 [ 1139.828853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_error -r 1 [ 1139.880286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_error -r 1 [ 1139.908693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_error -r 1 [ 1139.942601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_error -r 1 [ 1139.977425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_error -r 1 [ 1140.027745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1140.047848] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1140.337584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.370615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1140.402667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.439882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.475367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.542683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.583563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.657110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.702343] LNet: 62102:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.751599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.789978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.855571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.907767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.981902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.036845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.089491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1141.123861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1141.982343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_timeout -r 1 [ 1142.011501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_timeout -r 1 [ 1142.038583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_timeout -r 1 [ 1142.089824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_timeout -r 1 [ 1142.159181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_timeout -r 1 [ 1142.210190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_timeout -r 1 [ 1142.275033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_timeout -r 1 [ 1142.338644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_timeout -r 1 [ 1142.411843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.113@tcp [ 1142.439621] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1142.715903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.764256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1142.824571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.872650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.923703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.965368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.013451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.039377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1143.070205] LNet: 62225:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1143.099841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1143.159630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1143.211447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1143.269572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1143.375220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1143.418429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1143.480875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1143.532730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1149.484864] LNet: 62303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.491626] LNetError: 62303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.506248] LNet: Removed LNI 192.168.203.13@tcp [ 1149.534271] LNet: Removed LNI 192.168.203.13@tcp1 [ 1150.003642] Key type .llcrypt unregistered [ 1150.006131] Key type ._llcrypt unregistered [ 1166.687287] Key type ._llcrypt registered [ 1166.688937] Key type .llcrypt registered [ 1167.967084] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 04:48:14 (1780994894) [ 1169.891381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1172.040304] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 04:48:18 (1780994898) [ 1173.362288] Key type .llcrypt unregistered [ 1173.365476] Key type ._llcrypt unregistered [ 1180.499110] Key type ._llcrypt registered [ 1180.501556] Key type .llcrypt registered [ 1180.660295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1180.674457] alg: No test for adler32 (adler32-zlib) [ 1181.638519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1181.680137] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1190.789717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1191.535909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e network_timeout -r 1 [ 1191.584798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e network_timeout -r 1 [ 1191.627962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1191.660940] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1191.967059] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1192.040251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.104654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1192.162852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.249573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.301984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.358795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.414895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.526924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.583540] LNet: 63848:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.626512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.717281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.790775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.866622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.940513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.999171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.000581] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1193.429357] LNet: 63887:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.440340] LNetError: 63887:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.473466] LNet: Removed LNI 192.168.203.13@tcp [ 1195.020685] Key type .llcrypt unregistered [ 1195.027858] Key type ._llcrypt unregistered [ 1215.254503] Key type ._llcrypt registered [ 1215.255836] Key type .llcrypt registered [ 1215.515865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1215.543514] alg: No test for adler32 (adler32-zlib) [ 1216.479460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1216.513634] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1223.244918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1231.130596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.176696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1231.201324] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1232.290606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e network_timeout -r 1 [ 1232.317756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e network_timeout -r 1 [ 1232.352877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e network_timeout -r 1 [ 1232.391682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e network_timeout -r 1 [ 1232.423067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e network_timeout -r 1 [ 1232.456978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e network_timeout -r 1 [ 1232.501080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e network_timeout -r 1 [ 1232.555454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e network_timeout -r 1 [ 1232.609669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1232.638825] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1232.830461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1232.852916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1232.882193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1232.910604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1232.927111] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1232.929765] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 1232.942918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1232.977502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1233.031559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1233.077466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1233.100571] LNet: 64768:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1233.134422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1233.180785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1233.211950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1233.241742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1233.265766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1233.294753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.417287] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1237.723335] LNet: 64842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.730673] LNetError: 64842:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.737677] LNet: Removed LNI 192.168.203.13@tcp [ 1237.746200] LNet: Removed LNI 192.168.203.13@tcp1 [ 1238.063479] Key type .llcrypt unregistered [ 1238.067145] Key type ._llcrypt unregistered [ 1252.467823] Key type ._llcrypt registered [ 1252.470641] Key type .llcrypt registered [ 1253.784565] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 04:49:40 (1780994980) [ 1254.159826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1254.172551] alg: No test for adler32 (adler32-zlib) [ 1255.045600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1255.120535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1255.150474] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1255.946851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1255.964824] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1257.238849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1257.353706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1257.433671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1257.497834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1257.571714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1257.637072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1257.693165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.13@tcp [ 1257.721942] LNetError: 65941:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 1258.145900] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1258.151784] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1260.191383] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1261.362641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1264.293554] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1265.070469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1265.102745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1265.145229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1265.162995] LNet: 66048:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1265.168442] LNetError: 66048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.207149] LNet: Removed LNI 192.168.203.13@tcp [ 1266.215797] LNetError: 66048:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1266.230104] LNet: Removed LNI 192.168.203.13@tcp1 [ 1266.258889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.342772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1266.393626] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1267.444580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1267.487804] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1268.666892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1268.706492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.811961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1268.889472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1268.968032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1269.032470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1269.087728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1269.149954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.13@tcp [ 1269.210507] LNetError: 66321:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 1269.408235] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1271.653764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1275.553300] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1275.556900] LNet: Skipped 1 previous similar message [ 1280.778590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1280.818639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1280.868413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1287.911910] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 04:50:14 (1780995014) [ 1288.038668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.069571] LNet: 67036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.079782] LNetError: 67036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1288.102205] LNet: Removed LNI 192.168.203.13@tcp [ 1288.115615] LNet: Removed LNI 192.168.203.13@tcp1 [ 1288.139511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1288.209328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1288.238407] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1289.191220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1289.213757] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1290.532839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1290.658045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1290.695556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1290.729836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1290.756262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1290.780914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1290.817951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.13@tcp --health 0 [ 1290.896201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1291.232406] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.13@tcp [ 1291.237057] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1291.259124] LNet: Skipped 1 previous similar message [ 1306.447682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1306.625616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1306.664312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.699096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.13@tcp --health 500 [ 1306.794619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1313.651430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1313.711618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1313.733530] LNet: 67531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1313.742176] LNetError: 67531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1313.750609] LNetError: 67531:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1313.758903] LNet: Removed LNI 192.168.203.13@tcp [ 1313.768776] LNet: Removed LNI 192.168.203.13@tcp1 [ 1313.800437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.840299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1313.869615] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1314.693037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1314.722855] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1315.711641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1315.754610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.827793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1315.849854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1315.875968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1315.905255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1315.931379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1315.954536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.13@tcp --health 0 [ 1316.897615] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1316.905472] LNet: Skipped 3 previous similar messages [ 1317.217069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.500785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.533189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1326.569319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1332.878376] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 04:50:59 (1780995059) [ 1333.210271] LNet: 68567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.216050] LNetError: 68567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.224168] LNetError: 68567:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1333.231821] LNet: Removed LNI 192.168.203.13@tcp [ 1333.243712] LNet: Removed LNI 192.168.203.13@tcp1 [ 1333.787281] Key type .llcrypt unregistered [ 1333.794835] Key type ._llcrypt unregistered [ 1339.261741] Key type ._llcrypt registered [ 1339.264088] Key type .llcrypt registered [ 1339.397325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.410780] alg: No test for adler32 (adler32-zlib) [ 1340.330236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1340.363460] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1348.416502] Lustre: DEBUG MARKER: Initial discovery [ 1348.467439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.113@tcp [ 1350.486224] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1350.728427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1350.755827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.113@tcp [ 1350.793845] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1350.817220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1350.852905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.113@tcp -r 1 -m GET -l 3 [ 1350.874820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.113@tcp -r 1 -m GET -e local_error [ 1350.901723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.113@tcp -r 1 -m PUT -l 6 [ 1351.648952] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1352.071467] Lustre: DEBUG MARKER: Force oleg313-client.virtnet to discover 192.168.203.113@tcp (in background) [ 1352.298842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.113@tcp [ 1353.866720] Lustre: DEBUG MARKER: Force oleg313-server to discover 192.168.203.13@tcp [ 1355.360285] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1355.929556] Lustre: DEBUG MARKER: Wait for 69328 [ 1357.541120] Lustre: DEBUG MARKER: Finished wait on 69328 [ 1357.603823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1357.665912] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1357.679222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1358.004759] LNet: 69778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.013384] LNetError: 69778:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.039268] LNet: Removed LNI 192.168.203.13@tcp [ 1358.419990] Key type .llcrypt unregistered [ 1358.423535] Key type ._llcrypt unregistered [ 1375.371424] Key type ._llcrypt registered [ 1375.383503] Key type .llcrypt registered [ 1376.759776] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 04:51:43 (1780995103) [ 1377.275337] Key type .llcrypt unregistered [ 1377.279239] Key type ._llcrypt unregistered [ 1382.732635] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1382.918035] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1382.924598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1382.933471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1385.709734] Key type ._llcrypt registered [ 1385.712464] Key type .llcrypt registered [ 1385.933612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1385.944796] alg: No test for adler32 (adler32-zlib) [ 1386.911882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1386.967187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.004285] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1387.925798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1387.959503] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1397.095115] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 04:52:04 (1780995124) [ 1397.193641] LNet: 71846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.199587] LNetError: 71846:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.203839] LNet: Removed LNI 192.168.203.13@tcp [ 1397.216803] LNet: Removed LNI 10.1.2.3@tcp [ 1397.717625] Key type .llcrypt unregistered [ 1397.721736] Key type ._llcrypt unregistered [ 1404.200923] Key type ._llcrypt registered [ 1404.204612] Key type .llcrypt registered [ 1404.408374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1404.428591] alg: No test for adler32 (adler32-zlib) [ 1405.470374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1405.599324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1405.661320] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1406.318682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1406.333226] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1406.971448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.13@tcp [ 1408.032178] LNetError: Unexpected error -22 connecting to 192.168.203.13@tcp at host 192.168.203.13:988 [ 1413.123777] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 04:52:20 (1780995140) [ 1413.255477] LNet: 72977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.265957] LNetError: 72977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.280987] LNet: Removed LNI 192.168.203.13@tcp [ 1413.308265] LNet: Removed LNI 10.1.2.3@tcp [ 1413.903541] Key type .llcrypt unregistered [ 1413.913166] Key type ._llcrypt unregistered [ 1420.549787] Key type ._llcrypt registered [ 1420.554244] Key type .llcrypt registered [ 1420.840541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1420.882742] alg: No test for adler32 (adler32-zlib) [ 1421.766674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.807989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1421.836634] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1423.077746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1423.094296] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 1424.417213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.13@tcp1 --nid 192.168.203.13@tcp2 [ 1429.523710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp1 192.168.203.13@tcp2 [ 1429.575715] LNetError: 73820:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.13@tcp1 to 192.168.203.13@tcp2 - Route table may be misconfigured [ 1429.620493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp2 192.168.203.13@tcp1 [ 1437.769073] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 04:52:44 (1780995164) [ 1437.933959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1437.987099] LNet: 74387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.032201] LNetError: 74387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.071285] LNet: Removed LNI 192.168.203.13@tcp1 [ 1439.076858] LNetError: 74387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.116129] LNet: Removed LNI 192.168.203.13@tcp2 [ 1440.192827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.275676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1440.301593] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1441.327183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1441.356354] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1442.598859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 1442.707568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.13@tcp [ 1442.779800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp1 192.168.203.13@tcp1 [ 1442.874713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.13@tcp -d 192.168.203.13@tcp -e network_timeout [ 1442.947157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -e network_timeout [ 1443.023469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -e network_timeout [ 1443.097426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -e network_timeout [ 1443.166970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1443.188983] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1443.224525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1443.295187] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1443.391825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1443.566899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1450.683428] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 04:52:57 (1780995177) [ 1450.923410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.938200] LNet: 75257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.946286] LNetError: 75257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.958201] LNet: Removed LNI 192.168.203.13@tcp [ 1452.035495] LNet: Removed LNI 192.168.203.13@tcp1 [ 1452.052417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.108917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1452.330262] LNet: 75294:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.666561] Key type .llcrypt unregistered [ 1452.668944] Key type ._llcrypt unregistered [ 1462.218817] Key type ._llcrypt registered [ 1462.223474] Key type .llcrypt registered [ 1463.619761] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 04:53:10 (1780995190) [ 1464.098250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1464.113504] alg: No test for adler32 (adler32-zlib) [ 1465.025462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.272425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1467.951566] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 04:53:14 (1780995194) [ 1468.070781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.101497] LNet: 76388:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.158729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.246770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1468.276187] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1469.144883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1469.177887] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1470.181790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1470.217369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp1 [ 1470.245706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp1 [ 1477.913315] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 04:53:24 (1780995204) [ 1480.142255] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg313-server [ 1482.535279] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 04:53:28 (1780995208) [ 1485.122754] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg313-server [ 1487.439107] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 04:53:33 (1780995213) [ 1489.809273] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg313-server [ 1492.124502] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 04:53:38 (1780995218) [ 1494.890639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg313-server [ 1497.484740] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 04:53:43 (1780995223) [ 1499.721567] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg313-server [ 1501.514458] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 04:53:48 (1780995228) [ 1503.637393] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg313-server [ 1505.647118] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 04:53:52 (1780995232) [ 1507.838025] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg313-server [ 1509.578228] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 04:53:56 (1780995236) [ 1511.625116] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg313-server [ 1513.562167] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 04:54:00 (1780995240) [ 1515.589181] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg313-server [ 1516.961942] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 04:54:04 (1780995244) [ 1517.335206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.367287] LNet: 80270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1517.380790] LNetError: 80270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.387508] LNet: Removed LNI 192.168.203.13@tcp [ 1518.455705] LNet: Removed LNI 192.168.203.13@tcp1 [ 1518.516035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1518.590446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1518.628281] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1519.331875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1519.432466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1519.656322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.675944] LNet: 80417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.681493] LNetError: 80417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.684791] LNetError: 80417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1519.712201] LNet: Removed LNI 192.168.203.13@tcp [ 1519.739654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1519.809600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1519.861945] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1520.646636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1520.859141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.890518] LNetError: 80563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1521.963655] LNet: Removed LNI 192.168.203.13@tcp [ 1522.060791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.203880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1522.249841] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1523.289521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1523.434885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1523.757784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1523.784352] LNet: 80710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.792808] LNet: 80710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1523.802828] LNetError: 80710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1523.825745] LNet: Removed LNI 192.168.203.13@tcp [ 1523.851755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.910481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1523.943074] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1525.013180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1525.297985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.377585] LNet: Removed LNI 192.168.203.13@tcp [ 1526.427656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.607938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1526.643549] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1527.201707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1527.267435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1527.519150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.622245] LNet: Removed LNI 192.168.203.13@tcp [ 1528.672646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.855702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1528.909512] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1529.838473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1530.079162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.129617] LNet: 81148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.135520] LNet: 81148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1530.141372] LNetError: 81148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.151990] LNetError: 81148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1530.167643] LNet: Removed LNI 192.168.203.13@tcp [ 1530.184101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.227777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1530.270735] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1531.075451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1531.146141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1531.420205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1532.453221] LNet: Removed LNI 192.168.203.13@tcp [ 1532.509997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.641118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1532.685354] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1533.874933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1534.262452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.396921] LNet: Removed LNI 192.168.203.13@tcp [ 1535.436765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.536909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.574837] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1536.503597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1536.608302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1536.927685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.029427] LNet: Removed LNI 192.168.203.13@tcp [ 1538.134744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.214796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1538.255800] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1539.453868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1539.809264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.866455] LNet: 81734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.877125] LNet: 81734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1539.890978] LNetError: 81734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.906622] LNetError: 81734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1539.932960] LNet: Removed LNI 192.168.203.13@tcp [ 1539.976822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.048393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.087098] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1541.080515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1541.258374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1541.723821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.757783] LNet: Removed LNI 192.168.203.13@tcp [ 1542.848512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.955477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1543.014172] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1544.261314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1544.577904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.644539] LNet: Removed LNI 192.168.203.13@tcp [ 1545.686335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.822140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.869553] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1547.102086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1547.175714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.216601] LNet: Removed LNI 192.168.203.13@tcp [ 1547.234396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.296971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.313925] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1548.468814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1556.094505] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 04:54:42 (1780995282) [ 1556.231098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.277611] LNet: 82857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.285583] LNet: 82857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1556.312753] LNetError: 82857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.318552] LNetError: 82857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1556.338090] LNet: Removed LNI 192.168.203.13@tcp [ 1556.363943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.453475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1556.483837] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 1557.795592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1557.830593] LNet: Removed LNI 192.168.203.13@tcp231 [ 1557.850154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.934159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1557.987323] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 1557.993537] LNet: 83010:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1558.567421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1558.634302] LNet: Removed LNI 192.168.203.13@tcp231 [ 1558.673925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1558.705665] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 1560.237722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.261031] LNet: Removed LNI 192.168.203.13@tcp231 [ 1560.278421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.350389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1560.374260] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/0] [ 1560.381362] LNet: 83162:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1560.670695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1560.709312] LNet: Removed LNI 192.168.203.13@tcp231 [ 1560.734601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1560.793677] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/0] [ 1561.944305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.005518] LNet: Removed LNI 192.168.203.13@tcp231 [ 1562.025228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.123400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1562.141821] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/60] [ 1562.150127] LNet: 83314:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1562.730439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1562.775291] LNet: Removed LNI 192.168.203.13@tcp231 [ 1562.793452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1562.861053] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/60] [ 1563.797491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.818620] LNet: Removed LNI 192.168.203.13@tcp231 [ 1563.835122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.904189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1563.939853] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 1564.197171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1564.225556] LNet: Removed LNI 192.168.203.13@tcp231 [ 1564.248804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1564.286628] LNet: Added LNI 192.168.203.13@tcp231 [8/256/0/180] [ 1573.193722] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 04:55:00 (1780995300) [ 1573.381485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.453249] LNet: Removed LNI 192.168.203.13@tcp231 [ 1573.474495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.562596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1573.597792] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 1574.394872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1574.441184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1574.490622] LNet: Removed LNI 192.168.203.13@tcp232 [ 1574.517702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1574.558083] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 1575.354527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1575.401602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1575.445935] LNet: Removed LNI 192.168.203.13@tcp232 [ 1575.462279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1575.502612] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 1576.256866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1576.289620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1576.319412] LNet: Removed LNI 192.168.203.13@tcp232 [ 1576.343213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1576.362976] LNet: Added LNI 192.168.203.13@tcp232 [8/256/0/180] [ 1577.042520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1577.074754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1577.102685] LNet: Removed LNI 192.168.203.13@tcp232 [ 1577.121472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1582.988419] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 04:55:10 (1780995310) [ 1584.129300] Key type .llcrypt unregistered [ 1584.131563] Key type ._llcrypt unregistered [ 1589.554459] Key type ._llcrypt registered [ 1589.560465] Key type .llcrypt registered [ 1589.778752] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.794359] alg: No test for adler32 (adler32-zlib) [ 1590.730757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1590.764500] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1597.782447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1605.864960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.926308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1605.970442] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1607.640467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_interrupt -r 2 [ 1607.671374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 2 [ 1607.707771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_interrupt -r 2 [ 1607.766791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 2 [ 1607.815246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_interrupt -r 2 [ 1607.900511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_interrupt -r 2 [ 1607.982742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_interrupt -r 2 [ 1608.069853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_interrupt -r 2 [ 1608.168473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 1608.208413] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1608.222359] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1608.670847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1608.715241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1608.755181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1608.784637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1608.827689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1608.866499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1608.920470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1608.959490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1608.980577] LNet: 85965:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1609.001578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1609.038222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1609.073662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1609.105338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1609.145407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1609.187870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1609.892877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_dropped -r 2 [ 1609.929330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 2 [ 1609.988330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_dropped -r 2 [ 1610.031039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 2 [ 1610.123422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_dropped -r 2 [ 1610.191790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_dropped -r 2 [ 1610.231237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_dropped -r 2 [ 1610.269274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_dropped -r 2 [ 1610.316147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 1610.365093] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1610.676523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1610.733862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1610.783752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1610.826873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1610.883682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1610.935444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1610.975934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1611.039821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1611.068573] LNet: 86085:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1611.091485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1611.152275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1611.217073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1611.273305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1611.315430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1611.374671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1612.107334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_aborted -r 2 [ 1612.177552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 2 [ 1612.225510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_aborted -r 2 [ 1612.279328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 2 [ 1612.331287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_aborted -r 2 [ 1612.372979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_aborted -r 2 [ 1612.415967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_aborted -r 2 [ 1612.444792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_aborted -r 2 [ 1612.477175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 1612.494774] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1612.757295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1612.804072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1612.850792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1612.902407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1612.954192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1612.998910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1613.051203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1613.114965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1613.140633] LNet: 86205:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1613.169911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1613.211945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1613.260647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1613.317811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1613.378984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1613.438864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1614.138675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_no_route -r 2 [ 1614.186841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 2 [ 1614.245289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_no_route -r 2 [ 1614.299496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 2 [ 1614.375262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_no_route -r 2 [ 1614.467951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_no_route -r 2 [ 1614.511543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_no_route -r 2 [ 1614.546529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_no_route -r 2 [ 1614.572759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 1614.587484] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1614.853398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1614.880456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1614.909974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1614.961156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.003521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.033736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.063939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.103520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.150872] LNet: 86326:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1615.181890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1615.240177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1615.282727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.343364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1615.407703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1615.485812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.501707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e local_timeout -r 2 [ 1616.571203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 2 [ 1616.628724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e local_timeout -r 2 [ 1616.657426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 2 [ 1616.687188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e local_timeout -r 2 [ 1616.728363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e local_timeout -r 2 [ 1616.764633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e local_timeout -r 2 [ 1616.803229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e local_timeout -r 2 [ 1616.865534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 1617.351747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1617.401624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1617.472345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1617.523781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1617.593155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1617.661609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.707934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.761116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.820599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.885837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.923652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.948841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.974858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.015755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1618.650631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp -m GET -e remote_dropped -r 2 [ 1618.684407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 2 [ 1618.712220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp -m GET -e remote_dropped -r 2 [ 1618.782373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 2 [ 1618.849320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp -m GET -e remote_dropped -r 2 [ 1618.917564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.113@tcp1 -m GET -e remote_dropped -r 2 [ 1618.982536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp -m GET -e remote_dropped -r 2 [ 1619.042738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.13@tcp1 -d 192.168.203.13@tcp1 -m GET -e remote_dropped -r 2 [ 1619.090454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.13@tcp 192.168.203.113@tcp [ 1619.123280] LNet: There was an unexpected network error while writing to 192.168.203.113: rc = -22 [ 1619.126555] LNet: Skipped 1 previous similar message [ 1619.136209] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 1619.451890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1619.497871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1619.560700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1619.669060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1619.745894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1619.787451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1619.822758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1619.855104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1619.870505] LNet: 86565:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1619.874894] LNet: 86565:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1619.888465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1619.914579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1619.939944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1619.966902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1619.993764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1620.028628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1626.748783] LNet: 86640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.756107] LNetError: 86640:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.762789] LNet: Removed LNI 192.168.203.13@tcp [ 1626.775949] LNet: Removed LNI 192.168.203.13@tcp1 [ 1627.221720] Key type .llcrypt unregistered [ 1627.228182] Key type ._llcrypt unregistered [ 1646.255324] Key type ._llcrypt registered [ 1646.265182] Key type .llcrypt registered [ 1647.784912] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 04:56:14 (1780995374) [ 1648.287784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.309071] alg: No test for adler32 (adler32-zlib) [ 1649.255619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.304715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.336243] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1650.236920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1650.264463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.13@tcp [ 1650.287787] LNetError: 87619:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 1650.335405] LNet: 1 local NIs in recovery (showing 1): 192.168.203.13@tcp [ 1650.443686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.13@tcp [ 1650.583573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1657.265785] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 04:56:24 (1780995384) [ 1657.447972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.478181] LNet: 88204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.503346] LNetError: 88204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.545307] LNet: Removed LNI 192.168.203.13@tcp [ 1657.588484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.708587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.736762] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1658.268615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1658.284352] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1659.181461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.13@tcp [ 1659.195681] LNetError: 88474:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.13@tcp to recovery [ 1659.745496] LNet: There was an unexpected network error while writing to 192.168.203.13: rc = -22 [ 1660.506567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.13@tcp [ 1670.248429] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 04:56:36 (1780995396) [ 1672.095537] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg313-server [ 1673.935776] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 04:56:40 (1780995400) [ 1676.190494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg313-server [ 1678.609435] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 04:56:44 (1780995404) [ 1681.539775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg313-server [ 1684.006903] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 04:56:50 (1780995410) [ 1684.096039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.124374] LNet: 90097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.129629] LNetError: 90097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.151907] LNet: Removed LNI 192.168.203.13@tcp [ 1685.155086] LNetError: 90097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.174686] LNet: Removed LNI 192.168.203.13@tcp1 [ 1685.222438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.316770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1685.360449] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1686.125563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1686.170247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1686.200536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1686.260468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1686.304734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1686.336228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1694.199619] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 04:57:01 (1780995421) [ 1694.413015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.448858] LNet: 90832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1694.459265] LNetError: 90832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.474620] LNet: Removed LNI 192.168.203.13@tcp [ 1694.517544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.599880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.633639] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1701.868969] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 04:57:08 (1780995428) [ 1702.129369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.167140] LNet: 91538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1702.173448] LNetError: 91538:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1702.198189] LNet: Removed LNI 192.168.203.13@tcp [ 1702.225805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.281272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.316250] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1710.121143] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 04:57:16 (1780995436) [ 1711.893073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1713.642716] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 04:57:20 (1780995440) [ 1714.301765] LNet: 92575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.316816] LNetError: 92575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.344722] LNet: Removed LNI 192.168.203.13@tcp [ 1715.072799] Key type .llcrypt unregistered [ 1715.075269] Key type ._llcrypt unregistered [ 1721.663732] Key type ._llcrypt registered [ 1721.667087] Key type .llcrypt registered [ 1721.971983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1722.003906] alg: No test for adler32 (adler32-zlib) [ 1722.910852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1722.937424] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1731.408133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1738.582741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.113@tcp [ 1739.040552] LNet: 92993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.051266] LNetError: 92993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.071194] LNet: Removed LNI 192.168.203.13@tcp [ 1739.629875] Key type .llcrypt unregistered [ 1739.637738] Key type ._llcrypt unregistered [ 1751.925392] Key type ._llcrypt registered [ 1751.928051] Key type .llcrypt registered [ 1753.179412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1754.944375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1756.591416] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 04:58:03 (1780995483) [ 1758.179859] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1760.141266] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 04:58:06 (1780995486) [ 1762.118993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg313-server [ 1763.900096] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 04:58:10 (1780995490) [ 1765.651497] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg313-server [ 1767.499980] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 04:58:14 (1780995494) [ 1768.952181] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1768.967417] alg: No test for adler32 (adler32-zlib) [ 1770.023950] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 1770.336938] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1772.079171] Key type lgssc registered [ 1773.330639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1796.749385] Key type lgssc unregistered [ 1796.980712] LNet: 96170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.991220] LNetError: 96170:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1797.017463] LNet: Removed LNI 192.168.203.13@tcp [ 1797.785805] Key type .llcrypt unregistered [ 1797.790909] Key type ._llcrypt unregistered [ 1815.092477] Key type ._llcrypt registered [ 1815.095351] Key type .llcrypt registered [ 1816.358808] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 04:59:03 (1780995543) [ 1816.707199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1816.724471] alg: No test for adler32 (adler32-zlib) [ 1817.618160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.680833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1817.748149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1817.794251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1817.838542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1823.053301] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 04:59:10 (1780995550) [ 1823.181633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.217304] LNet: 97580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.258096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.339678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1823.370352] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1824.439369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.464513] LNet: 97729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.472429] LNetError: 97729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.489253] LNet: Removed LNI 192.168.203.13@tcp [ 1824.508827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.557794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1824.596072] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1824.940511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.977748] LNet: Removed LNI 192.168.203.13@tcp [ 1825.010626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.053322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1825.087343] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1831.516710] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 04:59:18 (1780995558) [ 1831.729792] LNet: 98347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.736581] LNet: 98347:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1831.745067] LNetError: 98347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.751933] LNetError: 98347:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1831.785204] LNet: Removed LNI 192.168.203.13@tcp [ 1832.414646] Key type .llcrypt unregistered [ 1832.416757] Key type ._llcrypt unregistered [ 1839.917379] Key type ._llcrypt registered [ 1839.920722] Key type .llcrypt registered [ 1840.160935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1840.193259] alg: No test for adler32 (adler32-zlib) [ 1841.049076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.158894] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1841.166354] LNet: *** cfs_fail_loc=e003, val=100*** [ 1842.225226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1842.247500] LNet: *** cfs_fail_loc=e003, val=100*** [ 1842.252134] LNet: Skipped 2 previous similar messages [ 1842.259415] Ping reply received with mapping: [ 1842.261162] found metadata reply NI[192.168.203.13@tcp] [ 1842.268703] 0d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1842.277510] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1842.282396] 5a 5a 5a 5a 5a 5a 5a 5a [ 1842.420277] LNet: 98746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.430713] LNetError: 98746:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.496666] LNet: Removed LNI 192.168.203.13@tcp [ 1844.272426] Key type .llcrypt unregistered [ 1844.274143] Key type ._llcrypt unregistered [ 1857.170398] Key type ._llcrypt registered [ 1857.171963] Key type .llcrypt registered [ 1858.598170] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 04:59:45 (1780995585) [ 1859.178782] Key type .llcrypt unregistered [ 1859.181051] Key type ._llcrypt unregistered [ 1866.539647] Key type ._llcrypt registered [ 1866.550148] Key type .llcrypt registered [ 1866.807161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1866.830159] alg: No test for adler32 (adler32-zlib) [ 1868.387395] LNetError: 99793:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1868.819932] Key type .llcrypt unregistered [ 1868.822959] Key type ._llcrypt unregistered [ 1874.798472] Key type ._llcrypt registered [ 1874.800954] Key type .llcrypt registered [ 1876.270107] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 05:00:02 (1780995602) [ 1876.874375] Key type .llcrypt unregistered [ 1876.877306] Key type ._llcrypt unregistered [ 1882.646137] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1882.650107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1885.300369] Key type ._llcrypt registered [ 1885.302137] Key type .llcrypt registered [ 1885.482559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1885.495835] alg: No test for adler32 (adler32-zlib) [ 1886.444591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.515410] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1895.646204] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 05:00:22 (1780995622) [ 1895.730985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.768860] LNet: 101568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.776228] LNetError: 101568:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1895.796127] LNet: Removed LNI 10.1.2.3@tcp [ 1895.833302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.519555] Key type .llcrypt unregistered [ 1896.524406] Key type ._llcrypt unregistered [ 1903.079471] Key type ._llcrypt registered [ 1903.081778] Key type .llcrypt registered [ 1903.230804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1903.257629] alg: No test for adler32 (adler32-zlib) [ 1904.233300] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1904.299213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1904.510294] LNet: 101863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.517032] LNetError: 101863:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.575923] LNet: Removed LNI 192.168.203.13@tcp [ 1906.214338] Key type .llcrypt unregistered [ 1906.221941] Key type ._llcrypt unregistered [ 1916.495422] Key type ._llcrypt registered [ 1916.499422] Key type .llcrypt registered [ 1917.736531] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 05:00:44 (1780995644) [ 1918.153366] Key type .llcrypt unregistered [ 1918.155653] Key type ._llcrypt unregistered [ 1925.846382] Key type ._llcrypt registered [ 1925.847751] Key type .llcrypt registered [ 1926.252574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1926.268649] alg: No test for adler32 (adler32-zlib) [ 1936.918412] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 05:01:03 (1780995663) [ 1937.036543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.097492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.200090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1937.256105] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1938.198842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.232454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1938.261387] LNet: Removed LNI 192.168.203.13@tcp [ 1938.504867] LNet: 103622:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.994837] Key type .llcrypt unregistered [ 1938.998446] Key type ._llcrypt unregistered [ 1950.956713] Key type ._llcrypt registered [ 1950.964919] Key type .llcrypt registered [ 1952.390851] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 05:01:19 (1780995679) [ 1952.778183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1952.796300] alg: No test for adler32 (adler32-zlib) [ 1953.773911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.826615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1953.844284] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1954.379623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 1954.404399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.13@tcp [ 1954.418678] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1954.422378] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1954.425418] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1954, 0, 0 [ 1954.429843] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1954.432434] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1954.435664] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1954.438570] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1954, 0, 0 [ 1954.442952] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1954.445604] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1954.448557] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1954.451678] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1954, 0, 2004 [ 1954.456171] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1954.458667] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1954.465896] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1954.468695] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1954, 0, 2004 [ 1954.472917] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1960.579240] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 05:01:27 (1780995687) [ 1961.302256] LNet: 105162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.308308] LNetError: 105162:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1961.325903] LNet: Removed LNI 192.168.203.13@tcp [ 1961.907839] Key type .llcrypt unregistered [ 1961.914174] Key type ._llcrypt unregistered [ 1968.564664] Key type ._llcrypt registered [ 1968.570910] Key type .llcrypt registered [ 1968.748654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1968.762582] alg: No test for adler32 (adler32-zlib) [ 1969.662830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1969.699772] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 1976.412533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 1987.655792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.694831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1987.730423] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 1988.849265] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1988.884843] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1988.888690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1988.891179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1991.342075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1991.361628] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2002.149418] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 05:02:08 (1780995728) [ 2002.251962] LNet: 106664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.264745] LNetError: 106664:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.271457] LNet: Removed LNI 192.168.203.13@tcp [ 2002.274677] LNet: Removed LNI 192.168.203.13@tcp1 [ 2002.305805] LNet: Removed LNI 10.1.2.3@tcp99 [ 2003.064804] Key type .llcrypt unregistered [ 2003.067050] Key type ._llcrypt unregistered [ 2009.097997] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2009.107616] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2011.676304] Key type ._llcrypt registered [ 2011.678562] Key type .llcrypt registered [ 2011.850295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2011.859719] alg: No test for adler32 (adler32-zlib) [ 2012.770471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.818896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2012.851517] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2013.531980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2013.554893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2015.492813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2015.540425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2015.567828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2015.587216] LNetError: 107417:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2015.611493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2015.648303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2016.050961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2016.121113] LNet: 107437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.126886] LNetError: 107437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2017.183166] LNet: Removed LNI 192.168.203.13@tcp [ 2017.187413] LNetError: 107437:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2017.207352] LNet: Removed LNI 10.1.2.3@tcp [ 2017.993626] Key type .llcrypt unregistered [ 2017.997682] Key type ._llcrypt unregistered [ 2033.099244] Key type ._llcrypt registered [ 2033.110885] Key type .llcrypt registered [ 2034.811538] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 05:02:41 (1780995761) [ 2035.224172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2035.238060] alg: No test for adler32 (adler32-zlib) [ 2036.235986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.279380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2036.297825] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2037.019141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.13@tcp [ 2037.103394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg313-client@tcp [ 2044.000736] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 05:02:50 (1780995770) [ 2044.157324] LNet: 108965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2044.165607] LNetError: 108965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2044.197466] LNet: Removed LNI 192.168.203.13@tcp [ 2044.678673] Key type .llcrypt unregistered [ 2044.680814] Key type ._llcrypt unregistered [ 2051.305756] Key type ._llcrypt registered [ 2051.310537] Key type .llcrypt registered [ 2051.444034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2051.458368] alg: No test for adler32 (adler32-zlib) [ 2052.444501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2052.472680] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2052.509374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2059.349919] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 05:03:06 (1780995786) [ 2060.713277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2062.728329] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 05:03:09 (1780995789) [ 2064.747724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2067.216470] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 05:03:13 (1780995793) [ 2067.375941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2067.415071] LNet: 110387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2067.424214] LNetError: 110387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2067.452699] LNet: Removed LNI 192.168.203.13@tcp [ 2067.517914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2067.626332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2067.653694] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2068.852710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2068.939239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2077.216214] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 05:03:23 (1780995803) [ 2077.338142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2077.378140] LNet: 111095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2077.392052] LNetError: 111095:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2077.419431] LNet: Removed LNI 192.168.203.13@tcp [ 2077.467681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.590225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2077.653695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2078.479722] Key type .llcrypt unregistered [ 2078.491881] Key type ._llcrypt unregistered [ 2092.557564] Key type ._llcrypt registered [ 2092.560140] Key type .llcrypt registered [ 2094.012591] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 05:03:40 (1780995820) [ 2094.563615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2094.582735] alg: No test for adler32 (adler32-zlib) [ 2095.536264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.682943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2095.706449] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2097.074231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2097.150742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2106.943530] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 05:03:52 (1780995832) [ 2107.052660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.077083] LNet: 112653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2107.093768] LNetError: 112653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2107.121221] LNet: Removed LNI 192.168.203.13@tcp [ 2107.150237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.223066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2107.262161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2117.137580] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 05:04:03 (1780995843) [ 2117.254231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2117.295040] LNet: 113232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2117.338941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.447924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2117.858702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2117.899858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2117.953093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2118.549287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2118.595787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2118.692977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.727204] LNet: 113272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2118.750636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2118.820578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2119.170447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2119.192393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2119.245354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2119.670931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2119.712833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2126.604080] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 05:04:13 (1780995853) [ 2126.716396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.748799] LNet: 113865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.778442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.879824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2126.930427] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2127.831739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2127.852826] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2128.847768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2128.930395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2129.177695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.206604] LNetError: 114151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.222935] LNet: Removed LNI 192.168.203.13@tcp [ 2129.242720] LNet: Removed LNI 192.168.203.13@tcp2 [ 2129.271113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.369208] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2129.376723] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2129.381242] LNet: 114157:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2129.956446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2130.168465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.231197] LNet: Removed LNI 192.168.203.13@tcp [ 2131.235548] LNetError: 114194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2131.243845] LNetError: 114194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2131.273500] LNet: Removed LNI 192.168.203.13@tcp2 [ 2131.309645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.479530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2131.532930] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2132.319518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2132.337251] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2133.241220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2133.315150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2133.388421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2133.694619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2133.727760] LNet: 114479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2133.735493] LNet: 114479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2133.747832] LNetError: 114479:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.754256] LNet: Removed LNI 192.168.203.13@tcp [ 2134.822661] LNet: Removed LNI 192.168.203.13@tcp2 [ 2134.856925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.964179] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2134.972534] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2134.980903] LNet: 114485:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2135.828196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2136.113482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.145654] LNet: Removed LNI 192.168.203.13@tcp [ 2137.192903] LNet: Removed LNI 192.168.203.13@tcp2 [ 2137.235926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.316908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.339854] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2138.065573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2138.109490] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2138.951454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2139.023228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2139.326603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.361493] LNetError: 114802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.377911] LNetError: 114802:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2139.388320] LNet: Removed LNI 192.168.203.13@tcp [ 2139.409107] LNet: Removed LNI 192.168.203.13@tcp2 [ 2139.433118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.520764] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2139.526813] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2139.533310] LNet: 114808:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2140.180610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2140.403793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.471146] LNet: Removed LNI 192.168.203.13@tcp [ 2141.486738] LNet: Removed LNI 192.168.203.13@tcp2 [ 2141.540277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2141.622691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2141.647591] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2142.379222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2142.422299] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2143.177944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2143.210962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2143.258666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2143.454858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.477747] LNet: 115129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.482601] LNet: 115129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2144.543267] LNet: Removed LNI 192.168.203.13@tcp [ 2145.587994] LNet: Removed LNI 192.168.203.13@tcp2 [ 2145.622466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.681904] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2145.688144] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2145.694650] LNet: 115136:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2146.248961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2146.511201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.536009] LNet: Removed LNI 192.168.203.13@tcp [ 2147.567504] LNet: Removed LNI 192.168.203.13@tcp2 [ 2147.634575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.697442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.719764] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2148.387327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2148.399421] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2149.290684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2149.364753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2149.428560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2149.803799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2149.832834] LNetError: 115469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2149.838065] LNetError: 115469:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2150.879261] LNet: Removed LNI 192.168.203.13@tcp [ 2150.891617] LNet: Removed LNI 192.168.203.13@tcp2 [ 2150.925324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.999724] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2151.007711] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2151.013983] LNet: 115475:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.809250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2152.262051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2153.311163] LNet: Removed LNI 192.168.203.13@tcp [ 2153.349513] LNet: Removed LNI 192.168.203.13@tcp2 [ 2153.383344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.483959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.511937] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2154.238932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2154.267090] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2155.300970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2155.344272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2155.397941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2155.450112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2155.815522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.895194] LNet: Removed LNI 192.168.203.13@tcp [ 2157.930717] LNet: Removed LNI 192.168.203.13@tcp2 [ 2157.969488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.041820] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2158.055965] LNet: Added LNI 192.168.203.13@tcp2 [8/256/0/180] [ 2158.613805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2166.184907] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2167.970537] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 05:04:54 (1780995894) [ 2168.133311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2168.169615] LNet: 116594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.180275] LNet: 116594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2168.189325] LNetError: 116594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.194218] LNetError: 116594:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2168.198139] LNet: Removed LNI 192.168.203.13@tcp [ 2169.267900] LNet: Removed LNI 192.168.203.13@tcp2 [ 2169.315293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.637530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2169.654765] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2170.332297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2170.346841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2171.180845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.203235] LNet: Removed LNI 192.168.203.13@tcp [ 2171.214206] LNet: Removed LNI 10.1.2.3@tcp [ 2171.231289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2171.303388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2171.323959] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2171.331660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2171.547247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2171.565685] LNet: Removed LNI 192.168.203.13@tcp [ 2171.574462] LNet: Removed LNI 10.1.2.3@tcp [ 2178.066749] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 05:05:04 (1780995904) [ 2178.184951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.226102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.412860] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2178.617532] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2178.620364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2178.623187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2181.013592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.067390] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2181.877382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2181.893810] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.302940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.13@tcp [ 2184.362778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2184.442838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.13@tcp -m PUT -e local_timeout -r 1 [ 2184.540591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2184.590522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2184.597979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2184.809224] LNetError: 117534:0:(lib-move.c:4714:lnet_parse()) 192.168.203.13@tcp, src 192.168.203.13@tcp: Dropping PUT (error -108 looking up sender) [ 2186.847277] LNet: Removed LNI 192.168.203.13@tcp [ 2186.857668] LNet: Removed LNI 10.1.2.3@tcp [ 2187.327333] Key type .llcrypt unregistered [ 2187.331638] Key type ._llcrypt unregistered [ 2198.803451] Key type ._llcrypt registered [ 2198.806494] Key type .llcrypt registered [ 2200.319714] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 05:05:26 (1780995926) [ 2200.822503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.844461] alg: No test for adler32 (adler32-zlib) [ 2201.818939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.179710] LNet: 118845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2203.031473] Key type .llcrypt unregistered [ 2203.036646] Key type ._llcrypt unregistered [ 2209.050422] Key type ._llcrypt registered [ 2209.056330] Key type .llcrypt registered [ 2209.414933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2209.445034] alg: No test for adler32 (adler32-zlib) [ 2210.421767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2210.473423] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2217.042635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 2217.259412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2217.911353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.113@tcp [ 2218.165553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2218.201178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2218.254925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2218.315571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2218.356451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2218.394656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2218.433836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2218.462674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2218.480846] LNet: 119294:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.501837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2218.540817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2218.577237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2218.628545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2218.656736] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 2218.675975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2218.719270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2219.381869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2219.713884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.783672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2219.830979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2219.899298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2219.947856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2219.982524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2220.025588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2220.068757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2220.087182] LNet: 119411:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.113285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2220.157585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2220.200788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2220.243803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2220.298936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2220.375375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.036988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.113@tcp [ 2221.259259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2221.306305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2221.345451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2221.424181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.478213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2221.532141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2221.572529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2221.619772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2221.657157] LNet: 119528:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2221.689866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2221.730218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2221.791562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2221.848924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2221.902508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.955275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2222.653727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2222.908400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2222.959470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2222.997985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2223.063097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2223.110282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2223.184708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2223.241395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2223.303778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2223.353578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2223.392690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2223.432215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2223.473717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2223.520420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2223.565730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2223.847967] LNet: 119691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.864346] LNetError: 119691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2223.905982] LNet: Removed LNI 192.168.203.13@tcp [ 2224.375781] Key type .llcrypt unregistered [ 2224.378580] Key type ._llcrypt unregistered [ 2241.063112] Key type ._llcrypt registered [ 2241.067898] Key type .llcrypt registered [ 2242.562897] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 05:06:09 (1780995969) [ 2242.918072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2242.933834] alg: No test for adler32 (adler32-zlib) [ 2243.855577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.297807] LNet: 120578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.819680] Key type .llcrypt unregistered [ 2244.824818] Key type ._llcrypt unregistered [ 2251.849369] Key type ._llcrypt registered [ 2251.857290] Key type .llcrypt registered [ 2252.083727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2252.127567] alg: No test for adler32 (adler32-zlib) [ 2253.116769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2253.146596] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2260.634614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.113@tcp [ 2270.900986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.998620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2271.033357] LNet: Added LNI 192.168.203.13@tcp1 [8/256/0/180] [ 2272.146550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2272.877932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.113@tcp [ 2273.178193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2273.202407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2273.247578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2273.279329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2273.329173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2273.379898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2273.434917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2273.495304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2273.524754] LNet: 121231:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2273.568503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2273.631459] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.113@tcp [ 2273.640487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2273.719933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2273.765264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2273.800753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2273.837874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.761099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2275.110974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2275.144182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2275.175740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2275.205844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2275.245729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2275.277609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2275.321944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2275.361540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2275.416665] LNet: 121348:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2275.464524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2275.539113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2275.584623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2275.616526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2275.659635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2275.728301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2276.588483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.113@tcp [ 2276.925557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2276.977957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2277.017243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2277.054418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2277.084634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2277.132765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2277.170902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2277.222523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2277.246749] LNet: 121465:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2277.279386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2277.307789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2277.341545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2277.388402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2277.451822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2277.517230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2278.316796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2278.652987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2278.696709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2278.746581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2278.820729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2278.879159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2278.939555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2278.988646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2279.049921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2279.078775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2279.124407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2279.157661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2279.187455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2279.237033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2279.291625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2285.077879] LNet: 121663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.089391] LNetError: 121663:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.104763] LNet: Removed LNI 192.168.203.13@tcp [ 2285.127667] LNet: Removed LNI 192.168.203.13@tcp1 [ 2285.635414] Key type .llcrypt unregistered [ 2285.641410] Key type ._llcrypt unregistered [ 2304.493549] Key type ._llcrypt registered [ 2304.496473] Key type .llcrypt registered [ 2305.963810] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 05:07:12 (1780996032) [ 2315.155762] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 05:07:21 (1780996041) [ 2323.400394] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 05:07:30 (1780996050) [ 2331.536994] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 05:07:38 (1780996058) [ 2338.653180] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 05:07:45 (1780996065) [ 2345.438708] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 05:07:52 (1780996072) [ 2353.033872] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 05:07:59 (1780996079) [ 2361.257986] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 05:08:07 (1780996087) [ 2368.851880] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 05:08:15 (1780996095) [ 2376.780995] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 05:08:23 (1780996103) [ 2385.406143] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 05:08:31 (1780996111) [ 2394.807577] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 05:08:40 (1780996120) [ 2403.475056] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 05:08:50 (1780996130) [ 2411.639471] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 05:08:58 (1780996138) [ 2419.388609] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 05:09:06 (1780996146) [ 2426.479879] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 05:09:13 (1780996153) [ 2433.662532] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 05:09:20 (1780996160) [ 2441.051358] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 05:09:27 (1780996167) [ 2441.395236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.408863] alg: No test for adler32 (adler32-zlib) [ 2442.314444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2442.441714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2442.465858] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2443.789659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2443.816958] LNet: 132376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2443.827648] LNetError: 132376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.845253] LNet: Removed LNI 192.168.203.13@tcp [ 2443.878623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.941882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2443.960601] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2450.814947] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 05:09:37 (1780996177) [ 2458.101698] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 05:09:45 (1780996185) [ 2466.939356] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 05:09:53 (1780996193) [ 2474.329729] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 05:10:01 (1780996201) [ 2480.457576] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 05:10:07 (1780996207) [ 2489.086460] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 05:10:15 (1780996215) [ 2498.328428] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 05:10:24 (1780996224) [ 2509.235883] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 05:10:35 (1780996235) [ 2518.577453] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 05:10:44 (1780996244) [ 2526.996455] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 05:10:53 (1780996253) [ 2535.538163] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 05:11:02 (1780996262) [ 2542.410737] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 05:11:09 (1780996269) [ 2548.598923] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2289 sec ========== 05:11:15 (1780996275) [ 2548.996646] LNet: 139841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2549.013554] LNetError: 139841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2549.042493] LNet: Removed LNI 192.168.203.13@tcp [ 2549.646632] Key type .llcrypt unregistered [ 2549.651691] Key type ._llcrypt unregistered [ 2562.524906] Key type ._llcrypt registered [ 2562.533338] Key type .llcrypt registered [ 2562.949235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2562.962908] alg: No test for adler32 (adler32-zlib) [ 2564.069574] Lustre: Lustre: Build Version: 2.17.54_1_g695b805 [ 2564.427797] LNet: Added LNI 192.168.203.13@tcp [8/256/0/180] [ 2566.135224] Key type lgssc registered [ 2567.353146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2628.007051] Lustre: Mounted lustre-client [ 2632.607729] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2640.796545] Lustre: Unmounted lustre-client [ 2666.300572] Key type lgssc unregistered [ 2666.562220] LNet: 142768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2666.570483] LNetError: 142768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2666.585203] LNet: Removed LNI 192.168.203.13@tcp [ 2667.214754] Key type .llcrypt unregistered [ 2667.219802] Key type ._llcrypt unregistered