[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452413888 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002378] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007363] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010131] LSM: Security Framework initializing [ 0.011071] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013082] *** VALIDATE selinux *** [ 0.021778] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026432] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027151] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029106] *** VALIDATE tmpfs *** [ 0.031365] *** VALIDATE proc *** [ 0.032264] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035268] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042367] debug: unmapping init [mem 0xffffffffa7e59000-0xffffffffa7e60fff] [ 0.044181] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049015] ... value mask: 0000ffffffffffff [ 0.050018] ... max period: 00007fffffffffff [ 0.051016] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053310] rcu: Hierarchical SRCU implementation. [ 0.055534] smp: Bringing up secondary CPUs ... [ 0.056617] x86: Booting SMP configuration: [ 0.057034] .... node #0, CPUs: #1 #2 #3 [ 0.060522] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.195029] node 0 deferred pages initialised in 130ms [ 0.198191] devtmpfs: initialized [ 0.200306] x86/mm: Memory block size: 128MB [ 0.202684] gcov: version magic: 0x41383552 [ 0.206402] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.209099] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.212265] pinctrl core: initialized pinctrl subsystem [ 0.214197] [ 0.214783] ************************************************************* [ 0.217015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220013] ** ** [ 0.222015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224023] ** ** [ 0.226011] ** This means that this kernel is built to expose internal ** [ 0.228014] ** IOMMU data structures, which may compromise security on ** [ 0.230017] ** your system. ** [ 0.233013] ** ** [ 0.235015] ** If you see this message and you are not debugging the ** [ 0.238019] ** kernel, report this immediately to your vendor! ** [ 0.240012] ** ** [ 0.243014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.245013] ************************************************************* [ 0.248554] NET: Registered protocol family 16 [ 0.250346] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.252061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.255066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.259171] cpuidle: using governor menu [ 0.260748] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.261571] PCI: Using configuration type 1 for base access [ 0.263121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.272121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.275055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.279089] cryptd: max_cpu_qlen set to 1000 [ 0.283122] ACPI: Added _OSI(Module Device) [ 0.284010] ACPI: Added _OSI(Processor Device) [ 0.285013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.287014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.291874] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.297028] ACPI: Interpreter enabled [ 0.298049] ACPI: PM: (supports S0 S3 S4 S5) [ 0.299023] ACPI: Using IOAPIC for interrupt routing [ 0.301113] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.315314] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.317040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.320023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.324085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.330761] acpiphp: Slot [2] registered [ 0.332185] acpiphp: Slot [5] registered [ 0.334191] acpiphp: Slot [6] registered [ 0.336204] acpiphp: Slot [3] registered [ 0.337256] acpiphp: Slot [4] registered [ 0.339164] acpiphp: Slot [7] registered [ 0.341155] acpiphp: Slot [8] registered [ 0.343117] acpiphp: Slot [9] registered [ 0.345144] acpiphp: Slot [10] registered [ 0.346176] acpiphp: Slot [11] registered [ 0.348117] acpiphp: Slot [12] registered [ 0.349141] acpiphp: Slot [13] registered [ 0.351140] acpiphp: Slot [14] registered [ 0.353130] acpiphp: Slot [15] registered [ 0.354181] acpiphp: Slot [16] registered [ 0.356156] acpiphp: Slot [17] registered [ 0.358138] acpiphp: Slot [18] registered [ 0.359124] acpiphp: Slot [19] registered [ 0.361128] acpiphp: Slot [20] registered [ 0.362142] acpiphp: Slot [21] registered [ 0.364129] acpiphp: Slot [22] registered [ 0.365160] acpiphp: Slot [23] registered [ 0.367124] acpiphp: Slot [24] registered [ 0.368127] acpiphp: Slot [25] registered [ 0.369120] acpiphp: Slot [26] registered [ 0.371129] acpiphp: Slot [27] registered [ 0.372117] acpiphp: Slot [28] registered [ 0.374138] acpiphp: Slot [29] registered [ 0.375139] acpiphp: Slot [30] registered [ 0.376109] acpiphp: Slot [31] registered [ 0.378096] PCI host bridge to bus 0000:00 [ 0.379021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.381026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.384037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.386041] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.389034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.392030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.393201] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.397027] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.400137] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.408017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.412043] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.415021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.417030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.420030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.422531] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.425992] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.429053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.431904] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.436000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.447019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.451018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.457178] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.469018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.492017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503821] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.527016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.536591] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.539393] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.541359] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.542558] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.545230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.551039] iommu: Default domain type: Passthrough [ 0.552361] SCSI subsystem initialized [ 0.554134] ACPI: bus type USB registered [ 0.555114] usbcore: registered new interface driver usbfs [ 0.556085] usbcore: registered new interface driver hub [ 0.557070] usbcore: registered new device driver usb [ 0.559179] pps_core: LinuxPPS API ver. 1 registered [ 0.561013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.564061] PTP clock support registered [ 0.567043] EDAC MC: Ver: 3.0.0 [ 0.568396] PCI: Using ACPI for IRQ routing [ 0.570769] NetLabel: Initializing [ 0.572013] NetLabel: domain hash size = 128 [ 0.574013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.576087] NetLabel: unlabeled traffic allowed by default [ 0.580095] vgaarb: loaded [ 0.582258] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.584017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.591000] clocksource: Switched to clocksource kvm-clock [ 0.691820] VFS: Disk quotas dquot_6.6.0 [ 0.693186] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.695146] *** VALIDATE ramfs *** [ 0.696479] *** VALIDATE hugetlbfs *** [ 0.698215] pnp: PnP ACPI init [ 0.700215] pnp: PnP ACPI: found 6 devices [ 0.718804] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.722738] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.724650] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.726883] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.729070] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.731436] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.734201] NET: Registered protocol family 2 [ 0.736536] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.741306] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.746263] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.751861] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.755349] TCP: Hash tables configured (established 65536 bind 65536) [ 0.759169] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.762380] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.764933] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.769179] NET: Registered protocol family 1 [ 0.771773] RPC: Registered named UNIX socket transport module. [ 0.774458] RPC: Registered udp transport module. [ 0.776385] RPC: Registered tcp transport module. [ 0.778314] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.780996] NET: Registered protocol family 44 [ 0.782850] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.785122] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.788704] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.791160] PCI: CLS 0 bytes, default 64 [ 0.793227] Unpacking initramfs... [ 2.242262] debug: unmapping init [mem 0xffff9c2d7cc64000-0xffff9c2d7ffcffff] [ 2.246564] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.249644] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.253020] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.763219] Initialise system trusted keyrings [ 2.765097] Key type blacklist registered [ 2.767820] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.778102] zbud: loaded [ 2.781366] *** VALIDATE nfs *** [ 2.782995] *** VALIDATE nfs4 *** [ 2.784907] pstore: using deflate compression [ 2.790313] Platform Keyring initialized [ 2.887602] NET: Registered protocol family 38 [ 2.889896] Key type asymmetric registered [ 2.891682] Asymmetric key parser 'x509' registered [ 2.894132] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.898400] io scheduler mq-deadline registered [ 2.900218] io scheduler kyber registered [ 2.901558] io scheduler bfq registered [ 2.902973] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.905978] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.909144] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.912060] ACPI: Power Button [PWRF] [ 2.918422] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.924238] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.933411] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.961967] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.992062] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.997114] Non-volatile memory driver v1.3 [ 2.998660] Linux agpgart interface v0.103 [ 3.030754] virtio_blk virtio1: [vda] 146728 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.033550] vda: detected capacity change from 0 to 75124736 [ 3.048713] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.052311] vdb: detected capacity change from 0 to 1073741824 [ 3.058185] libphy: Fixed MDIO Bus: probed [ 3.063253] usbcore: registered new interface driver usbserial_generic [ 3.065827] usbserial: USB Serial support registered for generic [ 3.068486] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.074351] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.076234] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.078701] mousedev: PS/2 mouse device common for all mice [ 3.081421] rtc_cmos 00:05: RTC can wake from S4 [ 3.084517] rtc_cmos 00:05: registered as rtc0 [ 3.087695] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.088798] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.090901] intel_pstate: CPU model not supported [ 3.096753] hid: raw HID events driver (C) Jiri Kosina [ 3.099439] usbcore: registered new interface driver usbhid [ 3.099805] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.101676] usbhid: USB HID core driver [ 3.107261] drop_monitor: Initializing network drop monitor service [ 3.109962] Initializing XFRM netlink socket [ 3.110059] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.111443] NET: Registered protocol family 10 [ 3.115854] Segment Routing with IPv6 [ 3.117698] NET: Registered protocol family 17 [ 3.119612] mpls_gso: MPLS GSO support [ 3.124753] RAS: Correctable Errors collector initialized. [ 3.127392] AVX version of gcm_enc/dec engaged. [ 3.128894] AES CTR mode by8 optimization enabled [ 3.204929] sched_clock: Marking stable (3204898349, 0)->(4157012296, -952113947) [ 3.208213] registered taskstats version 1 [ 3.210190] Loading compiled-in X.509 certificates [ 3.213520] zswap: loaded using pool lzo/zbud [ 3.239592] Key type big_key registered [ 3.251881] Key type encrypted registered [ 3.253671] ima: No TPM chip found, activating TPM-bypass! [ 3.255859] ima: Allocated hash algorithm: sha1 [ 3.257788] ima: No architecture policies found [ 3.259643] evm: Initialising EVM extended attributes: [ 3.261882] evm: security.selinux [ 3.263354] evm: security.ima [ 3.264517] evm: security.capability [ 3.265938] evm: HMAC attrs: 0x1 [ 3.268160] rtc_cmos 00:05: setting system clock to 2026-09-08 03:12:29 UTC (1788837149) [ 3.275341] debug: unmapping init [mem 0xffffffffa8e03000-0xffffffffa8ffffff] [ 3.278756] debug: unmapping init [mem 0xffffffffa7b82000-0xffffffffa7e58fff] [ 3.287097] Write protecting the kernel read-only data: 28672k [ 3.290364] debug: unmapping init [mem 0xffffffffa6203000-0xffffffffa63fffff] [ 3.293201] debug: unmapping init [mem 0xffffffffa6b14000-0xffffffffa6bfffff] [ 3.323104] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.332049] systemd[1]: Detected virtualization kvm. [ 3.334095] systemd[1]: Detected architecture x86-64. [ 3.335400] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.361821] systemd[1]: No hostname configured. [ 3.363842] systemd[1]: Set hostname to . [ 3.366308] random: systemd: uninitialized urandom read (16 bytes read) [ 3.369202] systemd[1]: Initializing machine ID from random generator. [ 3.491604] random: systemd: uninitialized urandom read (16 bytes read) [ 3.494184] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.497328] random: systemd: uninitialized urandom read (16 bytes read) [ 3.499777] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.504334] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.095054] device-mapper: uevent: version 1.0.3 [ 4.097541] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] [ 4.730857] random: fast init done Started Hardware RNG Entropy Gatherer Daemon. [ 4.747462] virtio_net virtio0 ens2: renamed from eth0 [ 4.763934] scsi host0: ata_piix [ 4.804714] scsi host1: ata_piix [ 4.812112] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.814739] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.452703] dracut-initqueue[579]: RTNETLINK answers: File exists [ 9.786381] random: crng init done [ 9.787997] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.056420] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.119992] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.364544] SELinux: Disabled at runtime. [ 11.426532] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.435736] systemd[1]: Detected virtualization kvm. [ 11.437991] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.907225] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.910844] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.916118] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.919618] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.923742] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.930336] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.938730] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibi[ 11.982779] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS lity Named Pipe. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.369285] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.622492] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.663314] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.744640] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.754937] EDAC sbridge: Ver: 1.1.2 [ 13.778558] Key type dns_resolver registered [ 14.080451] NFS: Registering the id_resolver key type [ 14.082378] Key type id_resolver registered [ 14.084071] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg353-client login: [ 52.991057] hrtimer: interrupt took 3066919 ns [ 78.427956] libcfs: loading out-of-tree module taints kernel. [ 78.499665] Key type ._llcrypt registered [ 78.504259] Key type .llcrypt registered [ 79.096855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 79.115486] alg: No test for adler32 (adler32-zlib) [ 80.615256] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 81.524605] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 83.250353] Key type lgssc registered [ 85.088808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 205.782000] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795 [ 210.623582] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.020212] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing check_logdir /tmp/testlogs/ [ 231.294169] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing yml_node [ 231.391247] Lustre: lustre-OST0000-osc-ffff9c2dc3107800: disconnect after 23s idle [ 235.673380] Lustre: DEBUG MARKER: Client: 2.17.57.113 [ 238.298928] Lustre: DEBUG MARKER: MDS: 2.17.57.113 [ 240.484570] Lustre: DEBUG MARKER: OSS: 2.17.57.113 [ 242.129215] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 23:16:27 EDT 2026 [ 260.296959] Lustre: Unmounted lustre-client [ 288.264922] Key type lgssc unregistered [ 288.474570] LNet: 8443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.494199] LNetError: 8443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.518065] LNet: Removed LNI 192.168.203.53@tcp [ 289.251134] Key type .llcrypt unregistered [ 289.252571] Key type ._llcrypt unregistered [ 318.654966] Key type ._llcrypt registered [ 318.664922] Key type .llcrypt registered [ 318.908322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 318.939171] alg: No test for adler32 (adler32-zlib) [ 319.946601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 320.062281] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 320.114321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 324.970582] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 325.435344] LNet: 9323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.455505] LNetError: 9323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 325.478627] LNet: Removed LNI 192.168.203.53@tcp [ 326.130132] Key type .llcrypt unregistered [ 326.134310] Key type ._llcrypt unregistered [ 333.156556] Key type ._llcrypt registered [ 333.162432] Key type .llcrypt registered [ 334.505141] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:17:59 (1788837479) [ 335.003699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 335.023924] alg: No test for adler32 (adler32-zlib) [ 335.962545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.049217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.076634] LNet: 9731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.106188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.146154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 336.165966] LNet: 9738:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 342.568387] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:18:07 (1788837487) [ 342.767372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.793160] LNet: 10316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 342.826609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.901061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 350.124099] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:18:15 (1788837495) [ 350.356293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.378083] LNet: 10902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.406981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.505029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 357.805218] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:18:22 (1788837502) [ 357.914389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.939241] LNet: 11482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 357.989702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.106781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 366.149798] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:18:30 (1788837510) [ 366.392071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.455350] LNet: 12072:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.496335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.593332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 367.106985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 374.822642] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:18:39 (1788837519) [ 375.245795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.274499] LNet: 12665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.306875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.408653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 376.022060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 376.645097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 384.286091] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:18:48 (1788837528) [ 384.560957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.596965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.650813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 392.449465] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:18:57 (1788837537) [ 392.877522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.924492] LNet: 13846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.937103] LNet: 13846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.985792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.075773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 393.121510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 393.774271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 393.836444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 394.319447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 394.365752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 394.771783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 394.834684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 395.579568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 395.636666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.261190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 396.322865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 396.942943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 397.027113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 397.704470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 397.749421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 398.387386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 398.438148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 399.079413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 399.130744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 399.728658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 399.778546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 400.230562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 400.286984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 400.812269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 400.877722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.357675] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:19:12 (1788837552) [ 408.473200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.561305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.713923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 408.791817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 416.346618] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:19:20 (1788837560) [ 416.532275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.627898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.749181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.809742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 423.854695] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:19:28 (1788837568) [ 423.952549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.010557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.122236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 424.185741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 430.930401] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:19:35 (1788837575) [ 431.002377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.028531] LNet: 16282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.034719] LNet: 16282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 431.063273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.187730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 431.240984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 438.175832] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:19:43 (1788837583) [ 438.366712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.430282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.497517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 438.539509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 445.370044] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:19:50 (1788837590) [ 445.554500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.624571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 445.733225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 445.774510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 451.887897] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:19:57 (1788837597) [ 452.185449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.245715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.317469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 452.553766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 458.056780] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:20:03 (1788837603) [ 458.417621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.468653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 458.641746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 458.892746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 465.624597] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:20:10 (1788837610) [ 465.901996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 465.955269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.184082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.451401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 474.279789] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:20:18 (1788837618) [ 474.379048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.436077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.261849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.495292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.722792] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:20:28 (1788837628) [ 483.861806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.937721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.110338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.182393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.618868] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:20:36 (1788837636) [ 491.877058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.922129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.026262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.074769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 499.106413] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:20:43 (1788837643) [ 499.228731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.256363] LNet: 21725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.262540] LNet: 21725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 499.292652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.402488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 507.065624] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:20:51 (1788837651) [ 507.173785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.224597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.359780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 514.671841] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:20:59 (1788837659) [ 514.976621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.057127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.187683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 515.951949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 523.772316] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:21:08 (1788837668) [ 524.074648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.127990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.262722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 524.321483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 531.482179] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:21:16 (1788837676) [ 532.017136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.078934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.170457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 532.259379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 540.162935] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:21:24 (1788837684) [ 540.331321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.401898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.558843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 540.625443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 548.443137] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:21:33 (1788837693) [ 548.799612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.859972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.999608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 549.042578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 557.031843] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:21:41 (1788837701) [ 557.189094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.260512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.338563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 557.436295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 557.466150] LNetError: 25850:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 557.547440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 557.582817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 564.671754] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:21:49 (1788837709) [ 564.742215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 564.788498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.946462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.dHxYngaVmJ [ 565.046115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.dHxYngaVmJ [ 565.126667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.dHxYngaVmJ [ 571.624973] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:21:56 (1788837716) [ 572.011356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.060229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.116933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 572.154964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 572.198642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 578.568702] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 23:22:03 (1788837723) [ 579.012446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 579.070122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.169739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 579.190381] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 580.860870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 582.544737] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 584.559168] Key type lgssc registered [ 586.020886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.208595] LNetError: 27612:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.53@tcp99 has no available nets [ 592.795937] Key type lgssc unregistered [ 593.167649] LNetError: 28717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.220263] LNet: Removed LNI 192.168.203.53@tcp [ 594.827333] Key type .llcrypt unregistered [ 594.835162] Key type ._llcrypt unregistered [ 602.105605] Key type ._llcrypt registered [ 602.112396] Key type .llcrypt registered [ 603.331664] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:22:28 (1788837748) [ 603.903125] Key type .llcrypt unregistered [ 603.905207] Key type ._llcrypt unregistered [ 610.894676] Key type ._llcrypt registered [ 610.897273] Key type .llcrypt registered [ 611.082638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 611.106994] alg: No test for adler32 (adler32-zlib) [ 612.097613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.159249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 612.176250] LNet: 29628:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 612.183482] LNet: 29628:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 612.190981] LNet: 29628:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 612.205619] LNet: Message forwarding will be enabled in 2s [ 615.263136] LNet: Message forwarding enabled [ 615.739130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 615.784817] LNet: Message forwarding will stop in 2s to 6s [ 619.361533] LNet: Message forwarding disabled [ 620.394742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 620.424723] LNet: 29726:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 620.447347] LNet: 29726:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 620.461761] LNet: 29726:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 620.498019] LNet: Message forwarding will be enabled in 2s [ 620.777386] LNet: 29739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.443483] Key type .llcrypt unregistered [ 621.446954] Key type ._llcrypt unregistered [ 634.980355] Key type ._llcrypt registered [ 634.985168] Key type .llcrypt registered [ 636.391401] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:23:00 (1788837780) [ 637.111304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 637.135769] alg: No test for adler32 (adler32-zlib) [ 638.116726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.217266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 644.542372] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:23:09 (1788837789) [ 644.798810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 644.849138] LNet: 31140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 644.872023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 644.954811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 645.040406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 645.084497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 645.122273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 645.153463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 645.187368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 645.228388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 645.267738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 645.315866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 645.346677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 645.395752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 645.437032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 645.478414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 645.543637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 645.583159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 645.625470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 645.671444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 645.738277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 645.801692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 645.824082] LNetError: 31183:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 652.803499] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:23:17 (1788837797) [ 653.123832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.151513] LNet: 31756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.183355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.316676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 660.909260] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:23:25 (1788837805) [ 663.078128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg353-server [ 664.705358] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:23:29 (1788837809) [ 664.932722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.979852] LNet: 32686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.006310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.129220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.165449] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 665.953883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[54-58]@tcp [ 665.982846] LNet: 32820:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 666.022685] LNetError: 32690:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.56@tcp is being used as a gateway but routing feature is not turned on [ 666.036337] LNetError: 32690:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.56@tcp has gone from up to down [ 673.354450] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:23:38 (1788837818) [ 675.417322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg353-server [ 677.032361] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:23:41 (1788837821) [ 677.262073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.303131] LNet: 33755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.311786] LNetError: 33755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.343778] LNet: Removed LNI 192.168.203.53@tcp [ 677.386151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.448154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.472893] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 678.311242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[54-59/2]@tcp [ 678.338842] LNet: 33873:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 678.348823] LNet: 33873:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 678.367792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[54-59/2]@tcp [ 678.391474] LNetError: 33760:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.56@tcp is being used as a gateway but routing feature is not turned on [ 678.416544] LNetError: 33760:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 678.434554] LNetError: 33760:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.56@tcp has gone from up to down [ 678.448896] LNetError: 33760:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 686.186364] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:23:50 (1788837830) [ 686.481576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.501211] LNet: 34465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.518313] LNetError: 34465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 687.591413] LNet: Removed LNI 192.168.203.53@tcp [ 687.634406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.709731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 687.773352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.818262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.914234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 687.994138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.049145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.124740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 688.234700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.296651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.349717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.426700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.471330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.535459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 688.653702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.690991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.772557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.852778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.888969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.946521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 689.051544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.111699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.226331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.313542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.393586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.458446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 689.558090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.605159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.710914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 689.865461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.911599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.952883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 689.996815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.030098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.096738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.670285] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:24:02 (1788837842) [ 699.669455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg353-server [ 701.231878] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:24:06 (1788837846) [ 703.235355] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg353-server [ 705.144840] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:24:09 (1788837849) [ 705.242219] LNet: 35857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.250095] LNet: 35857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 705.816685] Key type .llcrypt unregistered [ 705.822286] Key type ._llcrypt unregistered [ 710.982908] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 711.064906] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.074063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 711.086519] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 712.541985] Key type ._llcrypt registered [ 712.545710] Key type .llcrypt registered [ 712.762655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 712.792509] alg: No test for adler32 (adler32-zlib) [ 713.745519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.830353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.871681] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 714.503957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 714.524457] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 716.576822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 716.595700] LNet: Removed LNI 10.1.2.3@tcp [ 716.725361] LNet: 36577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.736652] LNetError: 36577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.760612] LNet: Removed LNI 192.168.203.53@tcp [ 717.339386] Key type .llcrypt unregistered [ 717.341564] Key type ._llcrypt unregistered [ 728.105712] Key type ._llcrypt registered [ 728.107417] Key type .llcrypt registered [ 729.376321] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:24:34 (1788837874) [ 729.812380] Key type .llcrypt unregistered [ 729.817206] Key type ._llcrypt unregistered [ 735.784675] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 735.980573] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 735.990542] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 735.998625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 738.544550] Key type ._llcrypt registered [ 738.552560] Key type .llcrypt registered [ 738.743338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 738.758870] alg: No test for adler32 (adler32-zlib) [ 739.706625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 739.777946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 739.803493] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 740.582213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.604777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 743.393097] LNet: 38065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.407457] LNetError: 38065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 743.417212] LNet: Removed LNI 10.1.2.3@tcp [ 743.435430] LNet: Removed LNI 192.168.203.53@tcp [ 744.124973] Key type .llcrypt unregistered [ 744.130196] Key type ._llcrypt unregistered [ 756.819361] Key type ._llcrypt registered [ 756.821895] Key type .llcrypt registered [ 758.481899] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:25:02 (1788837902) [ 758.995801] Key type .llcrypt unregistered [ 758.998233] Key type ._llcrypt unregistered [ 766.002237] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.161796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.183398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.194864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 768.795351] Key type ._llcrypt registered [ 768.797768] Key type .llcrypt registered [ 768.978376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.000817] alg: No test for adler32 (adler32-zlib) [ 769.935582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.014970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 770.065857] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 772.016115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 772.036112] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 773.994021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 774.024487] LNet: Removed LNI 10.1.2.3@tcp [ 774.049290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 774.090820] LNet: Removed LNI 10.1.2.31@tcp [ 774.115642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.134265] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.943510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 774.972449] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 775.931430] LNet: 39975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.947260] LNetError: 39975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 775.952825] LNet: Removed LNI 10.1.2.31@tcp [ 775.971686] LNet: Removed LNI 10.1.2.3@tcp [ 776.517592] Key type .llcrypt unregistered [ 776.520193] Key type ._llcrypt unregistered [ 789.865800] Key type ._llcrypt registered [ 789.871674] Key type .llcrypt registered [ 791.339521] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:25:35 (1788837935) [ 791.749605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 791.763400] alg: No test for adler32 (adler32-zlib) [ 792.747224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.830235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 792.865453] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 794.127598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 794.170310] LNet: 40924:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.181255] LNetError: 40924:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.221964] LNet: Removed LNI 192.168.203.53@tcp [ 794.275174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.343164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.53@tcp [ 794.364123] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 795.318747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.53@tcp [ 795.357878] LNet: Removed LNI 192.168.203.53@tcp [ 795.477738] LNet: 41059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.361423] Key type .llcrypt unregistered [ 796.364444] Key type ._llcrypt unregistered [ 808.519665] Key type ._llcrypt registered [ 808.522902] Key type .llcrypt registered [ 809.810640] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:25:54 (1788837954) [ 811.894252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg353-server [ 813.544734] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:25:58 (1788837958) [ 814.085785] Key type .llcrypt unregistered [ 814.088164] Key type ._llcrypt unregistered [ 820.983144] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.078562] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.106164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.241449] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.212378] Key type ._llcrypt registered [ 824.216832] Key type .llcrypt registered [ 824.656059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.701184] alg: No test for adler32 (adler32-zlib) [ 825.846289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.020779] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 827.305928] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 829.595984] LNet: 42889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.602652] LNetError: 42889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.609280] LNet: Removed LNI 192.168.203.53@tcp1 [ 829.623043] LNet: Removed LNI 10.1.2.3@tcp2 [ 830.313347] Key type .llcrypt unregistered [ 830.315668] Key type ._llcrypt unregistered [ 846.041143] Key type ._llcrypt registered [ 846.042810] Key type .llcrypt registered [ 847.384646] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:26:32 (1788837992) [ 847.952691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.992482] alg: No test for adler32 (adler32-zlib) [ 848.910458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.961240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.984459] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 849.727574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.749107] LNet: 43856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.757739] LNetError: 43856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.781812] LNet: Removed LNI 192.168.203.53@tcp [ 849.806927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.886345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 849.936452] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 856.373694] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 23:26:41 (1788838001) [ 856.668928] LNet: 44447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.689100] LNetError: 44447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.707216] LNet: Removed LNI 192.168.203.53@tcp [ 857.326731] Key type .llcrypt unregistered [ 857.328524] Key type ._llcrypt unregistered [ 864.313487] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 864.404844] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.406376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.411297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.884342] Key type ._llcrypt registered [ 866.888254] Key type .llcrypt registered [ 867.179213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 867.194771] alg: No test for adler32 (adler32-zlib) [ 868.148334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.206391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.275528] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 869.091867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 869.132261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.650579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.674363] LNet: 45184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.687844] LNetError: 45184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 871.695270] LNet: Removed LNI 10.1.2.3@tcp [ 871.715558] LNet: Removed LNI 192.168.203.53@tcp [ 871.760186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.847428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 871.900462] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 871.918201] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.522949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.544935] LNet: 45211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.556688] LNetError: 45211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 872.560946] LNetError: 45211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 872.569777] LNet: Removed LNI 10.1.2.3@tcp [ 872.580707] LNet: Removed LNI 192.168.203.53@tcp [ 872.598883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.655956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 872.678873] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 872.683841] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 873.276720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 873.306265] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 874.484678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.506772] LNet: 45373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.510911] LNetError: 45373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.520639] LNetError: 45373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 874.527399] LNet: Removed LNI 10.1.2.3@tcp [ 874.529644] LNet: Removed LNI 192.168.203.53@tcp [ 874.549736] LNet: Removed LNI 192.168.203.53@tcp2 [ 874.575687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.675945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 874.709238] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 874.718746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 874.724824] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 882.987587] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 23:27:07 (1788838027) [ 883.236345] LNet: 45969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.244591] LNetError: 45969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.250335] LNetError: 45969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 883.259793] LNet: Removed LNI 10.1.2.3@tcp [ 883.262756] LNet: Removed LNI 192.168.203.53@tcp [ 883.286978] LNet: Removed LNI 192.168.203.53@tcp2 [ 883.934315] Key type .llcrypt unregistered [ 883.942431] Key type ._llcrypt unregistered [ 890.779464] Key type ._llcrypt registered [ 890.782864] Key type .llcrypt registered [ 891.049228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.099576] alg: No test for adler32 (adler32-zlib) [ 892.068571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.118199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.153347] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 892.902708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.925439] LNet: 46361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.934686] LNetError: 46361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.957609] LNet: Removed LNI 192.168.203.53@tcp [ 892.973312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.033303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.053244] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 893.547940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.563267] LNet: 46391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.566693] LNetError: 46391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.578734] LNet: Removed LNI 192.168.203.53@tcp [ 893.611307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.670458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.693397] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 893.969639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.991822] LNet: Removed LNI 192.168.203.53@tcp [ 894.009566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.069771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 894.098338] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 894.602764] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 894.800108] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 894.823392] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 894.834410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 897.246665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.258738] LNet: 46499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.263162] LNet: 46499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 897.272209] LNetError: 46499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.279486] LNetError: 46499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 897.297300] LNet: Removed LNI 192.168.203.53@tcp [ 897.314654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.387186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 897.430947] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 904.823117] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 23:27:29 (1788838049) [ 905.044710] LNet: 47098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.055900] LNetError: 47098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.070506] LNet: Removed LNI 192.168.203.53@tcp [ 905.555984] Key type .llcrypt unregistered [ 905.557827] Key type ._llcrypt unregistered [ 911.860265] Key type ._llcrypt registered [ 911.863570] Key type .llcrypt registered [ 912.048305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.066445] alg: No test for adler32 (adler32-zlib) [ 913.007435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.117313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.153330] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 914.011064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.031762] LNet: 47493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.036189] LNetError: 47493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.054633] LNet: Removed LNI 192.168.203.53@tcp [ 914.071495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.116222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 914.134106] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 921.496556] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 23:27:46 (1788838066) [ 921.842424] LNet: 48085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.858051] LNetError: 48085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.880429] LNet: Removed LNI 192.168.203.53@tcp [ 922.715404] Key type .llcrypt unregistered [ 922.724731] Key type ._llcrypt unregistered [ 929.658923] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 929.794374] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 929.800585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 929.812611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 932.576518] Key type ._llcrypt registered [ 932.579315] Key type .llcrypt registered [ 932.769552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 932.794899] alg: No test for adler32 (adler32-zlib) [ 933.727467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.784451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 933.809984] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 934.688588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 934.711652] LNet: 48530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.727478] LNetError: 48530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.740265] LNet: Removed LNI 192.168.203.53@tcp [ 934.778659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.849536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 934.871889] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 941.234342] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 23:28:06 (1788838086) [ 941.599487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.634322] LNet: 49128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.643756] LNetError: 49128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.687291] LNet: Removed LNI 192.168.203.53@tcp [ 941.716480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.851620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 948.451737] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 23:28:13 (1788838093) [ 948.748402] LNet: 49711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.749648] Key type .llcrypt unregistered [ 949.752827] Key type ._llcrypt unregistered [ 959.246860] Key type ._llcrypt registered [ 959.249098] Key type .llcrypt registered [ 959.537365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.560142] alg: No test for adler32 (adler32-zlib) [ 960.623046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.718523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.766956] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 961.820834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.877499] LNet: 50106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.889942] LNetError: 50106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.906819] LNet: Removed LNI 192.168.203.53@tcp [ 961.958433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.157413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 962.231174] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 970.424533] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 23:28:34 (1788838114) [ 970.782241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.809322] LNet: 50699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.814578] LNetError: 50699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.832931] LNet: Removed LNI 192.168.203.53@tcp [ 970.856622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.958989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 977.676860] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 23:28:42 (1788838122) [ 979.364426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 981.253863] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 23:28:46 (1788838126) [ 982.919893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 984.642383] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 23:28:49 (1788838129) [ 986.138670] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 988.100918] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 23:28:53 (1788838133) [ 989.468620] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 991.172552] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 23:28:56 (1788838136) [ 992.767728] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 994.657968] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 23:28:59 (1788838139) [ 995.086803] LNet: 52751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 995.835570] Key type .llcrypt unregistered [ 995.840531] Key type ._llcrypt unregistered [ 1003.100652] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1003.299585] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1003.314817] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1003.318344] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1005.783714] Key type ._llcrypt registered [ 1005.785528] Key type .llcrypt registered [ 1005.954342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1005.966497] alg: No test for adler32 (adler32-zlib) [ 1006.887979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1006.945587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1006.960052] LNet: Added LNI 192.168.203.53@tcp [8/512/0/180] [ 1006.963811] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1006.967621] LNet: 53069:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.044651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.058786] LNet: 53088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.064301] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.069624] LNet: Removed LNI 10.1.2.3@tcp [ 1007.072498] LNetError: 53088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.085233] LNet: Removed LNI 192.168.203.53@tcp [ 1007.101364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.150567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.171944] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1007.227269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.244198] LNet: 53112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.255315] LNetError: 53112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.276556] LNet: Removed LNI 192.168.203.53@tcp [ 1007.296994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.366982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.390460] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1007.398982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.403979] LNet: 53120:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.507358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.529984] LNet: 53138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.536079] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.541196] LNet: Removed LNI 10.1.2.3@tcp [ 1007.544418] LNetError: 53138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.558863] LNet: Removed LNI 192.168.203.53@tcp [ 1007.580574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1007.634414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1007.664499] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1007.672254] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.677855] LNet: 53146:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1007.932747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1007.962746] LNet: 53172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.968857] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.984421] LNet: Removed LNI 10.1.2.3@tcp [ 1007.988370] LNetError: 53172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.022557] LNet: Removed LNI 192.168.203.53@tcp [ 1008.062881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.129969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1008.158264] LNet: Added LNI 192.168.203.53@tcp [8/256/0/182] [ 1008.169791] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1008.282829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.301524] LNet: 53200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1008.306644] LNetError: 53200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1008.311903] LNet: Removed LNI 10.1.2.3@tcp [ 1008.313962] LNetError: 53200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1008.325751] LNet: Removed LNI 192.168.203.53@tcp [ 1008.341893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.443895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1008.461417] LNet: Added LNI 192.168.203.53@tcp [8/256/0/182] [ 1008.466552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1008.470395] LNet: 53208:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1008.526780] LNet: Removed LNI 10.1.2.3@tcp [ 1008.553943] LNet: Removed LNI 192.168.203.53@tcp [ 1009.201449] Key type .llcrypt unregistered [ 1009.207759] Key type ._llcrypt unregistered [ 1020.888789] Key type ._llcrypt registered [ 1020.892714] Key type .llcrypt registered [ 1022.232322] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 23:29:26 (1788838166) [ 1023.653658] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1025.788557] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:29:30 (1788838170) [ 1026.197785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1026.225494] alg: No test for adler32 (adler32-zlib) [ 1027.173152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.245715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.283927] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1028.133100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.53 [ 1028.149307] LNet: 54471:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1028.169821] LNet: Removed LNI 192.168.203.53@tcp [ 1033.677652] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:29:38 (1788838178) [ 1033.869631] LNet: 55037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.881267] LNetError: 55037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.898162] LNet: Removed LNI 192.168.203.53@tcp [ 1034.491623] Key type .llcrypt unregistered [ 1034.494422] Key type ._llcrypt unregistered [ 1040.009934] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1040.282735] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1042.989772] Key type ._llcrypt registered [ 1042.993763] Key type .llcrypt registered [ 1043.198804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1043.220849] alg: No test for adler32 (adler32-zlib) [ 1044.601526] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1049.912773] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:29:55 (1788838195) [ 1050.090418] LNet: 55950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1050.098810] LNetError: 55950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1050.128540] LNet: Removed LNI 10.1.2.3@tcp [ 1050.670965] Key type .llcrypt unregistered [ 1050.677215] Key type ._llcrypt unregistered [ 1055.654739] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1055.816710] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1055.825359] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1055.828127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.476524] Key type ._llcrypt registered [ 1058.481179] Key type .llcrypt registered [ 1058.606503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.619305] alg: No test for adler32 (adler32-zlib) [ 1059.571335] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1066.016206] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:30:10 (1788838210) [ 1067.499697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1068.880532] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:30:13 (1788838213) [ 1069.266232] LNet: 57147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.277782] LNetError: 57147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1069.297313] LNet: Removed LNI 10.1.2.3@tcp [ 1069.929426] Key type .llcrypt unregistered [ 1069.931053] Key type ._llcrypt unregistered [ 1075.847508] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1076.082697] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1076.088506] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1076.094219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1078.849750] Key type ._llcrypt registered [ 1078.851757] Key type .llcrypt registered [ 1079.128488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.145584] alg: No test for adler32 (adler32-zlib) [ 1080.068125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.134659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1088.485155] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:30:33 (1788838233) [ 1088.927739] LNet: 58373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.939488] LNetError: 58373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1088.950837] LNet: Removed LNI 10.1.2.3@tcp [ 1089.747444] Key type .llcrypt unregistered [ 1089.749708] Key type ._llcrypt unregistered [ 1096.228334] Key type ._llcrypt registered [ 1096.231119] Key type .llcrypt registered [ 1096.508123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.523680] alg: No test for adler32 (adler32-zlib) [ 1097.430452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1097.465555] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1105.359077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1106.254304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_interrupt -r 1 [ 1106.331976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_interrupt -r 1 [ 1106.421384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1106.441645] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1106.720120] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1106.794982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.850712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.888332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.936575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.978829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.030889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.106110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.161488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.191393] LNet: 58809:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.232591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.292197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.340052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.380633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.440536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.503787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1107.544895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1108.298206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_dropped -r 1 [ 1108.351494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_dropped -r 1 [ 1108.395217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1108.416165] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1108.713266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1108.745291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1108.780672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.813993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.847735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.879700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.917161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.961874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.994929] LNet: 58918:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.023661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.063218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.138886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.183836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.216757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.261681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.309261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.963305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_aborted -r 1 [ 1110.002745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_aborted -r 1 [ 1110.035925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1110.060987] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1110.311320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.345648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.378603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.450787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.490295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.543801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.612462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.674291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.689988] LNet: 59027:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.704537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.742972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.780615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.824939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.855226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.883754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.911342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.427736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_no_route -r 1 [ 1111.461584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_no_route -r 1 [ 1111.502507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1111.763606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.791629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1111.824500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.860182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.917380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.952068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.000844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.066684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.129218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.156832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.219960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.254331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.300048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.357493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.408685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.016961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_timeout -r 1 [ 1113.065514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_timeout -r 1 [ 1113.112142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1113.143606] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1113.155785] LNet: Skipped 2 previous similar messages [ 1113.380320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.418411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1113.482476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.529359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.575805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.617253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.658831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.727093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.749865] LNet: 59246:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.768764] LNet: 59246:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1113.800792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.865819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.899400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.932395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.973601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.008774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.048897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.621330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_error -r 1 [ 1114.668319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_error -r 1 [ 1114.713465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1114.989835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.027773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.077653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.123415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.173247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.218708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.264668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.308551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.343840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.384485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.427621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.463868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.486294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.519186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.549244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.842305] LNet: 59396:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1115.850889] LNetError: 59396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.873907] LNet: Removed LNI 192.168.203.53@tcp [ 1116.411406] Key type .llcrypt unregistered [ 1116.415798] Key type ._llcrypt unregistered [ 1135.444673] Key type ._llcrypt registered [ 1135.448344] Key type .llcrypt registered [ 1136.620668] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:31:21 (1788838281) [ 1137.497456] Key type .llcrypt unregistered [ 1137.499845] Key type ._llcrypt unregistered [ 1144.261328] Key type ._llcrypt registered [ 1144.264466] Key type .llcrypt registered [ 1144.569699] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.607714] alg: No test for adler32 (adler32-zlib) [ 1145.605859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1145.664616] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1153.777308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1166.298080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1166.390068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1166.426977] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1168.602918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_interrupt -r 1 [ 1168.642668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 1 [ 1168.679903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_interrupt -r 1 [ 1168.727466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 1 [ 1168.799835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_interrupt -r 1 [ 1168.855926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 1 [ 1168.907748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_interrupt -r 1 [ 1168.951813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 1 [ 1169.006675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1169.029960] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1169.038304] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1169.345564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.379433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.439612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.480738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.522288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.555433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.603122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.661625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.703388] LNet: 60862:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.748145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.789366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.855734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.912489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.981368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.048949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1170.086789] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1170.110300] LNet: Skipped 3 previous similar messages [ 1170.140449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1170.841708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_dropped -r 1 [ 1170.876456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 1 [ 1170.908830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_dropped -r 1 [ 1170.940247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 1 [ 1170.990438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_dropped -r 1 [ 1171.050202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 1 [ 1171.131816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_dropped -r 1 [ 1171.185018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 1 [ 1171.227334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1171.251200] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1171.455861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.491952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.519692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.562807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.596754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.623723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.651535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.684546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.710279] LNet: 60983:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.729653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.761569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.797624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.835772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.872767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.940453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1172.022825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.648214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_aborted -r 1 [ 1172.707481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 1 [ 1172.742709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_aborted -r 1 [ 1172.785408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 1 [ 1172.817661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_aborted -r 1 [ 1172.866779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 1 [ 1172.922672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_aborted -r 1 [ 1172.981899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 1 [ 1173.025851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1173.296389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.334818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.372644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.413655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.445592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.482874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.535885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.597641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.619682] LNet: 61105:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.638361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.694158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.742334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.782645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.838972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.881659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.947107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.730851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_no_route -r 1 [ 1174.786652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 1 [ 1174.850315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_no_route -r 1 [ 1174.901503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 1 [ 1174.959690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_no_route -r 1 [ 1174.998694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 1 [ 1175.049773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_no_route -r 1 [ 1175.166432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 1 [ 1175.239115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1175.290249] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1175.303532] LNet: Skipped 5 previous similar messages [ 1175.642642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.678896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.704803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.750454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.825561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.887457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.951562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.000344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.031197] LNet: 61226:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.054530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.087631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.137057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.171542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.220777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.279306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.334766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.013490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_timeout -r 1 [ 1177.066081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 1 [ 1177.103679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_timeout -r 1 [ 1177.145916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 1 [ 1177.186883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_timeout -r 1 [ 1177.257514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 1 [ 1177.300512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_timeout -r 1 [ 1177.360480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 1 [ 1177.406897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1177.674775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1177.719330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1177.748276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.772589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.801348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.820239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.846165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.875974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.907268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.947694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.978559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.022349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.070504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.158883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.204099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.882339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_error -r 1 [ 1178.967868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_error -r 1 [ 1179.038911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_error -r 1 [ 1179.084630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_error -r 1 [ 1179.119528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_error -r 1 [ 1179.149769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_error -r 1 [ 1179.185423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_error -r 1 [ 1179.238657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_error -r 1 [ 1179.294588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1179.319501] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1179.332946] LNet: Skipped 5 previous similar messages [ 1179.680698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.739211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.797456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.826195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.873226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.914748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.971605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.024213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.040183] LNet: 61469:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.043884] LNet: 61469:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1180.082384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.129457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.188516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.240844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.270808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.307032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.350087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1186.985167] LNet: 61545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1186.990935] LNetError: 61545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1187.006357] LNet: Removed LNI 192.168.203.53@tcp [ 1188.070702] LNet: Removed LNI 192.168.203.53@tcp1 [ 1188.363292] Key type .llcrypt unregistered [ 1188.367315] Key type ._llcrypt unregistered [ 1204.452208] Key type ._llcrypt registered [ 1204.455285] Key type .llcrypt registered [ 1205.953190] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:32:30 (1788838350) [ 1206.691280] Key type .llcrypt unregistered [ 1206.694969] Key type ._llcrypt unregistered [ 1213.155520] Key type ._llcrypt registered [ 1213.159311] Key type .llcrypt registered [ 1213.466541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.500244] alg: No test for adler32 (adler32-zlib) [ 1214.493532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.543968] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1222.756647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1223.808851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_dropped -r 1 [ 1223.898444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_dropped -r 1 [ 1223.953729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1223.982282] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1224.329310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1224.372288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1224.417617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.483398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.536450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.586345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.646677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.705582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.731845] LNet: 62790:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.748633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.795188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.834523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.894248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.945809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.002746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.054547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1225.819554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_error -r 1 [ 1225.905695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_error -r 1 [ 1225.993185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1226.028532] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1226.340167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1226.380862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1226.405987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1226.434514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1226.477407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1226.518120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1226.557624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1226.626171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1226.650885] LNet: 62899:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1226.663730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1226.700970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1226.764564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1226.800414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1226.856804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1226.900401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1226.951917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1227.607504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_timeout -r 1 [ 1227.639512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_timeout -r 1 [ 1227.694380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1227.731285] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1228.047228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1228.109944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1228.142888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1228.179976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1228.205856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1228.235541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1228.257844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1228.304476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1228.332159] LNet: 63008:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1228.362804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1228.411026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1228.450599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1228.513165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.580245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1228.618136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1228.675661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1229.048568] LNet: 63049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1229.070369] LNetError: 63049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.117684] LNet: Removed LNI 192.168.203.53@tcp [ 1229.641517] Key type .llcrypt unregistered [ 1229.643871] Key type ._llcrypt unregistered [ 1248.487668] Key type ._llcrypt registered [ 1248.490880] Key type .llcrypt registered [ 1250.304893] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:33:14 (1788838394) [ 1251.411850] Key type .llcrypt unregistered [ 1251.418348] Key type ._llcrypt unregistered [ 1259.843239] Key type ._llcrypt registered [ 1259.848591] Key type .llcrypt registered [ 1260.315364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1260.349377] alg: No test for adler32 (adler32-zlib) [ 1261.481153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1261.524151] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1270.464921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1280.178986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1280.205393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1280.218988] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1281.173048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_dropped -r 1 [ 1281.205035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 1 [ 1281.249901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_dropped -r 1 [ 1281.285498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 1 [ 1281.330089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_dropped -r 1 [ 1281.367880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 1 [ 1281.411138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_dropped -r 1 [ 1281.445634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 1 [ 1281.496081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1281.528242] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1281.540824] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 1281.834994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1281.885697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1281.938827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1281.991831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1282.048622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1282.093971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1282.128062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1282.164965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1282.179781] LNet: 64515:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1282.195922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1282.236220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1282.282530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1282.318755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1282.346919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.364678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1282.397701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1282.431287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1282.906680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_error -r 1 [ 1282.950835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_error -r 1 [ 1283.001822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_error -r 1 [ 1283.056719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_error -r 1 [ 1283.108551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_error -r 1 [ 1283.160597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_error -r 1 [ 1283.210075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_error -r 1 [ 1283.250988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_error -r 1 [ 1283.288463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1283.314731] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1283.327523] LNet: Skipped 2 previous similar messages [ 1283.574255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1283.608352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1283.653546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1283.691703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.732618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.791122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.815937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.840546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.850957] LNet: 64638:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1283.870690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.910809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.941994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.967988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1284.010146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.042793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1284.079338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1284.121802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1284.720642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_timeout -r 1 [ 1284.751575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_timeout -r 1 [ 1284.786347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_timeout -r 1 [ 1284.818723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_timeout -r 1 [ 1284.860494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_timeout -r 1 [ 1284.918617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_timeout -r 1 [ 1284.967510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_timeout -r 1 [ 1284.999283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_timeout -r 1 [ 1285.026926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1285.040795] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1285.216688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1285.242607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1285.265805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.294455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.339625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.377728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1285.431622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1285.485695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.503838] LNet: 64761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1285.517542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.538540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.566294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.590909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.621701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.666530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.692604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1285.728335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.538814] LNet: 64839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.547909] LNetError: 64839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1290.560081] LNet: Removed LNI 192.168.203.53@tcp [ 1290.578419] LNet: Removed LNI 192.168.203.53@tcp1 [ 1290.949990] Key type .llcrypt unregistered [ 1290.953708] Key type ._llcrypt unregistered [ 1308.010698] Key type ._llcrypt registered [ 1308.014378] Key type .llcrypt registered [ 1309.511698] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:34:14 (1788838454) [ 1311.189577] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1312.613445] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:34:17 (1788838457) [ 1313.506328] Key type .llcrypt unregistered [ 1313.508180] Key type ._llcrypt unregistered [ 1319.493562] Key type ._llcrypt registered [ 1319.497206] Key type .llcrypt registered [ 1319.814775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1319.843711] alg: No test for adler32 (adler32-zlib) [ 1320.824075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1320.901776] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1329.362587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1330.106944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e network_timeout -r 1 [ 1330.148578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e network_timeout -r 1 [ 1330.198225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1330.222457] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1330.523527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.573343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.636866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.696862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.728558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.759247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.790514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.819682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.838515] LNet: 66378:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.864588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.894611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.936719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.983514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1331.016687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1331.061744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1331.410375] LNet: 66417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1331.418702] LNetError: 66417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1331.436768] LNet: Removed LNI 192.168.203.53@tcp [ 1331.988684] Key type .llcrypt unregistered [ 1331.993742] Key type ._llcrypt unregistered [ 1350.912418] Key type ._llcrypt registered [ 1350.915660] Key type .llcrypt registered [ 1351.245208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1351.263771] alg: No test for adler32 (adler32-zlib) [ 1352.128086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1352.144121] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1361.061667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1373.053759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.103880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1373.154374] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1374.801702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e network_timeout -r 1 [ 1374.827780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e network_timeout -r 1 [ 1374.859737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e network_timeout -r 1 [ 1374.888291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e network_timeout -r 1 [ 1374.940763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e network_timeout -r 1 [ 1374.991468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e network_timeout -r 1 [ 1375.057449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e network_timeout -r 1 [ 1375.142629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e network_timeout -r 1 [ 1375.205298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1375.236904] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1375.599981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1375.645987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1375.706356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1375.712103] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1375.717487] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 1375.752079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1375.830803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1375.884531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1375.923499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1375.957207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1375.992892] LNet: 67294:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1376.020033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1376.057041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1376.089133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1376.146346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1376.182807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1376.213859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1382.282188] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1382.646083] LNet: 67368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1382.658503] LNetError: 67368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1382.668688] LNet: Removed LNI 192.168.203.53@tcp [ 1382.691852] LNet: Removed LNI 192.168.203.53@tcp1 [ 1383.071641] Key type .llcrypt unregistered [ 1383.074637] Key type ._llcrypt unregistered [ 1400.123174] Key type ._llcrypt registered [ 1400.126447] Key type .llcrypt registered [ 1401.373365] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:35:46 (1788838546) [ 1401.856082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1401.870675] alg: No test for adler32 (adler32-zlib) [ 1402.823893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.885191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.926410] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1403.671783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1403.702355] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1404.366812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1404.423713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1404.458429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1404.485537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1404.511889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1404.547379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1404.585078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.53@tcp [ 1404.623601] LNetError: 68466:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1404.895467] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1404.897346] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1406.944116] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1408.061551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1410.016927] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1410.454480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1410.483949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1410.532844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.549868] LNet: 68563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.556312] LNetError: 68563:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.562297] LNet: Removed LNI 192.168.203.53@tcp [ 1410.571973] LNet: Removed LNI 192.168.203.53@tcp1 [ 1410.589626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.644677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.679854] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1411.340335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.359078] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1412.111148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1412.154947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1412.216907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1412.275981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1412.327506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1412.378726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1412.407496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1412.430359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.53@tcp [ 1412.444169] LNetError: 68835:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1412.639816] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1414.825887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1418.786119] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1418.794595] LNet: Skipped 1 previous similar message [ 1423.905536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1423.934955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1423.975830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1430.426875] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:36:15 (1788838575) [ 1430.851966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.871667] LNet: 69553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.876698] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.883030] LNetError: 69553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1430.899534] LNet: Removed LNI 192.168.203.53@tcp [ 1430.910568] LNet: Removed LNI 192.168.203.53@tcp1 [ 1430.947708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.013343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.029850] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1431.676763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1431.695396] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1432.362506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1432.415981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1432.451746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1432.484782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1432.518813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1432.554580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1432.585553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.53@tcp --health 0 [ 1432.707873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1433.055389] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.53@tcp [ 1433.055566] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1433.073446] LNet: Skipped 1 previous similar message [ 1447.953073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1448.029713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1448.050398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1448.092934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.53@tcp --health 500 [ 1448.191488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1454.856484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1454.934293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.987882] LNet: 70045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.996506] LNetError: 70045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.014183] LNetError: 70045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1455.023727] LNet: Removed LNI 192.168.203.53@tcp [ 1455.047736] LNet: Removed LNI 192.168.203.53@tcp1 [ 1455.096346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.146160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.188095] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1456.079894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1456.100097] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1457.508432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1457.532326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1457.603815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1457.659476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1457.707251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1457.756781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1457.788169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1457.830951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.53@tcp --health 0 [ 1458.211669] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1458.216852] LNet: Skipped 3 previous similar messages [ 1459.422676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.777434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.812560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1469.843758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1476.721188] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:37:01 (1788838621) [ 1477.209355] LNet: 71091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1477.214424] LNetError: 71091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.219977] LNetError: 71091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1477.225956] LNet: Removed LNI 192.168.203.53@tcp [ 1477.237985] LNet: Removed LNI 192.168.203.53@tcp1 [ 1477.821608] Key type .llcrypt unregistered [ 1477.827536] Key type ._llcrypt unregistered [ 1485.959404] Key type ._llcrypt registered [ 1485.966973] Key type .llcrypt registered [ 1486.225129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1486.269694] alg: No test for adler32 (adler32-zlib) [ 1487.254482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1487.301279] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1496.325875] Lustre: DEBUG MARKER: Initial discovery [ 1496.468129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.153@tcp [ 1498.489490] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1498.710925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1498.756037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.153@tcp [ 1498.780899] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1498.797704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1498.831718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.153@tcp -r 1 -m GET -l 3 [ 1498.876951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.153@tcp -r 1 -m GET -e local_error [ 1498.920316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.153@tcp -r 1 -m PUT -l 6 [ 1499.615477] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1500.110823] Lustre: DEBUG MARKER: Force oleg353-client.virtnet to discover 192.168.203.153@tcp (in background) [ 1500.397582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.153@tcp [ 1501.559730] Lustre: DEBUG MARKER: Force oleg353-server to discover 192.168.203.53@tcp [ 1503.455331] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1503.860540] Lustre: DEBUG MARKER: Wait for 71852 [ 1505.271429] Lustre: DEBUG MARKER: Finished wait on 71852 [ 1505.532025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1505.587909] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1505.592098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1505.835721] LNet: 72300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.843863] LNetError: 72300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.858629] LNet: Removed LNI 192.168.203.53@tcp [ 1506.229940] Key type .llcrypt unregistered [ 1506.231588] Key type ._llcrypt unregistered [ 1521.241600] Key type ._llcrypt registered [ 1521.247163] Key type .llcrypt registered [ 1522.568977] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:37:47 (1788838667) [ 1523.190396] Key type .llcrypt unregistered [ 1523.192650] Key type ._llcrypt unregistered [ 1530.424274] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1530.571065] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1530.573897] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1530.576734] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1533.120636] Key type ._llcrypt registered [ 1533.123989] Key type .llcrypt registered [ 1533.313507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1533.330513] alg: No test for adler32 (adler32-zlib) [ 1534.277489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1534.329233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1534.349069] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1534.960280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1534.978311] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1543.576729] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:38:08 (1788838688) [ 1543.782751] LNet: 74368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.797869] LNetError: 74368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1543.808345] LNet: Removed LNI 10.1.2.3@tcp [ 1543.825332] LNet: Removed LNI 192.168.203.53@tcp [ 1544.452608] Key type .llcrypt unregistered [ 1544.456331] Key type ._llcrypt unregistered [ 1550.696212] Key type ._llcrypt registered [ 1550.698863] Key type .llcrypt registered [ 1550.857318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1550.869324] alg: No test for adler32 (adler32-zlib) [ 1551.827912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.885938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1551.919316] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1552.728975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1552.759742] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1553.751669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.53@tcp [ 1554.847115] LNetError: Unexpected error -22 connecting to 192.168.203.53@tcp at host 192.168.203.53:988 [ 1560.540871] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:38:25 (1788838705) [ 1560.643132] LNet: 75489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.653408] LNetError: 75489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1560.659519] LNet: Removed LNI 10.1.2.3@tcp [ 1560.687567] LNet: Removed LNI 192.168.203.53@tcp [ 1561.749158] Key type .llcrypt unregistered [ 1561.753824] Key type ._llcrypt unregistered [ 1568.993795] Key type ._llcrypt registered [ 1568.998180] Key type .llcrypt registered [ 1569.237365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1569.254769] alg: No test for adler32 (adler32-zlib) [ 1570.275403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.381623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1570.416913] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1571.265599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1571.290825] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 1572.421947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.53@tcp1 --nid 192.168.203.53@tcp2 [ 1577.859995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp1 192.168.203.53@tcp2 [ 1577.903477] LNetError: 76329:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.53@tcp1 to 192.168.203.53@tcp2 - Route table may be misconfigured [ 1577.931766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp2 192.168.203.53@tcp1 [ 1585.928385] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:38:50 (1788838730) [ 1586.013893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.055517] LNet: 76897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.068224] LNetError: 76897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.075103] LNet: Removed LNI 192.168.203.53@tcp1 [ 1587.113183] LNet: Removed LNI 192.168.203.53@tcp2 [ 1587.131566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.193903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1587.217666] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1588.175493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1588.210631] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1589.362778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1589.436883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.53@tcp [ 1589.474587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp1 192.168.203.53@tcp1 [ 1589.537528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.53@tcp -d 192.168.203.53@tcp -e network_timeout [ 1589.607768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -e network_timeout [ 1589.679922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -e network_timeout [ 1589.733172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -e network_timeout [ 1589.785719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1589.810131] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1589.828610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1589.929511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1590.029638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1590.239587] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1596.114165] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:39:01 (1788838741) [ 1596.487802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.542401] LNet: 77767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.559467] LNetError: 77767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.571389] LNetError: 77767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1596.585116] LNet: Removed LNI 192.168.203.53@tcp [ 1597.608834] LNet: Removed LNI 192.168.203.53@tcp1 [ 1597.673886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.822499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1598.074409] LNet: 77804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.409637] Key type .llcrypt unregistered [ 1598.414361] Key type ._llcrypt unregistered [ 1608.882571] Key type ._llcrypt registered [ 1608.884814] Key type .llcrypt registered [ 1610.155568] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:39:14 (1788838754) [ 1610.644385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1610.669502] alg: No test for adler32 (adler32-zlib) [ 1611.550269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.972486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1615.049885] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:39:19 (1788838759) [ 1615.157807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.194159] LNet: 78893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.225594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.299558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.317607] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1615.952746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1615.982895] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1616.900412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1616.946310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp1 [ 1616.991334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp1 [ 1622.157738] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:39:27 (1788838767) [ 1623.831810] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg353-server [ 1625.574164] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:39:30 (1788838770) [ 1627.684825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg353-server [ 1629.333576] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:39:34 (1788838774) [ 1631.001770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg353-server [ 1632.607804] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:39:37 (1788838777) [ 1634.390452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg353-server [ 1635.854061] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:39:40 (1788838780) [ 1637.385461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg353-server [ 1638.983992] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:39:44 (1788838784) [ 1640.517625] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg353-server [ 1641.872911] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:39:47 (1788838787) [ 1643.832221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg353-server [ 1645.320195] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:39:50 (1788838790) [ 1647.057824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg353-server [ 1648.412107] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:39:53 (1788838793) [ 1649.991095] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg353-server [ 1651.617143] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:39:56 (1788838796) [ 1651.718790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.759454] LNet: 82769:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.768419] LNetError: 82769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1651.780725] LNet: Removed LNI 192.168.203.53@tcp [ 1652.840262] LNet: Removed LNI 192.168.203.53@tcp1 [ 1652.869207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.922281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.940951] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1653.662623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1653.739883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1653.931162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.955169] LNet: 82923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.961745] LNetError: 82923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.966840] LNetError: 82923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1653.980656] LNet: Removed LNI 192.168.203.53@tcp [ 1653.999353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.062261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1654.097553] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1654.896157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1655.074304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.101078] LNetError: 83068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.110839] LNet: Removed LNI 192.168.203.53@tcp [ 1655.132149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.182723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.210337] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1655.881629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1655.957229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1656.136703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.166302] LNet: 83215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.173335] LNet: 83215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1657.192955] LNet: Removed LNI 192.168.203.53@tcp [ 1657.217357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.261888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1657.299686] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1658.175406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1658.347955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.380960] LNetError: 83361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.390410] LNetError: 83361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1658.403866] LNet: Removed LNI 192.168.203.53@tcp [ 1658.426715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.482482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.517392] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1659.109797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1659.213489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1659.435431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.519092] LNet: Removed LNI 192.168.203.53@tcp [ 1660.561831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.628262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1660.660161] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1661.485784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1661.788249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.806846] LNet: 83653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.813313] LNet: 83653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1662.889091] LNet: Removed LNI 192.168.203.53@tcp [ 1662.934211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.012490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1663.048102] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1663.824969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1663.942768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1664.260637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.289409] LNetError: 83800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.300807] LNetError: 83800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1664.316902] LNet: Removed LNI 192.168.203.53@tcp [ 1664.332665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.369577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1664.388290] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1665.293808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1665.553354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.662255] LNet: Removed LNI 192.168.203.53@tcp [ 1666.718367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.829645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1666.890414] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1667.819189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1667.899471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1668.159452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.224480] LNet: Removed LNI 192.168.203.53@tcp [ 1669.241994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.310775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1669.346234] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1670.399767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1670.804223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.839635] LNet: 84238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.850875] LNet: 84238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1671.910707] LNet: Removed LNI 192.168.203.53@tcp [ 1671.999634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.090357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1672.158418] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1673.090913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1673.148501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1673.309832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1673.339826] LNetError: 84386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.343906] LNetError: 84386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1673.358906] LNet: Removed LNI 192.168.203.53@tcp [ 1673.375224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.473477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1673.506954] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1674.225552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1674.546313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.637980] LNet: Removed LNI 192.168.203.53@tcp [ 1675.676202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.760980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.789280] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1676.690685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1676.819564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.915319] LNet: Removed LNI 192.168.203.53@tcp [ 1676.956581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.035545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1677.055071] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1677.947978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1678.084440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.146512] LNet: Removed LNI 192.168.203.53@tcp [ 1678.191910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.256538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1678.383804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.473494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.546157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1678.640865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.690786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.759298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1678.899828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.948978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.045541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1679.089508] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1679.821173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.862837] LNet: Removed LNI 192.168.203.53@tcp [ 1679.888694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.978898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1687.443491] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:40:32 (1788838832) [ 1687.766933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.779204] LNet: 85552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.782952] LNet: 85552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1687.795304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.826905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1687.843719] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1688.700448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.743121] LNet: Removed LNI 192.168.203.53@tcp231 [ 1688.761078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.835447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1688.869447] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1688.881183] LNet: 85698:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1689.227247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1689.257883] LNet: Removed LNI 192.168.203.53@tcp231 [ 1689.268863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1689.286596] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1690.105624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.143878] LNetError: 85842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1690.153626] LNetError: 85842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1690.169309] LNet: Removed LNI 192.168.203.53@tcp231 [ 1690.192039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.290688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1690.358243] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/0] [ 1690.370268] LNet: 85850:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1691.013622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1691.064907] LNet: Removed LNI 192.168.203.53@tcp231 [ 1691.087772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1691.130368] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/0] [ 1692.816072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.856159] LNet: Removed LNI 192.168.203.53@tcp231 [ 1692.886840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.988058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1693.024969] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/60] [ 1693.030299] LNet: 86002:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.483843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1693.510386] LNet: Removed LNI 192.168.203.53@tcp231 [ 1693.529792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1693.556798] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/60] [ 1695.040586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.081783] LNet: Removed LNI 192.168.203.53@tcp231 [ 1695.128135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.263970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1695.307326] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1695.317741] LNet: 86153:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.831271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1695.872856] LNet: Removed LNI 192.168.203.53@tcp231 [ 1695.897797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1695.941376] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1702.978574] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:40:48 (1788838848) [ 1703.087817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.149639] LNet: Removed LNI 192.168.203.53@tcp231 [ 1703.219304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.312994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1703.333926] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1704.109961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1704.145815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1704.181277] LNet: Removed LNI 192.168.203.53@tcp232 [ 1704.205746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1704.237175] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1705.060638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1705.105486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1705.136533] LNet: Removed LNI 192.168.203.53@tcp232 [ 1705.163110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1705.197113] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1706.022609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1706.076516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1706.118464] LNet: Removed LNI 192.168.203.53@tcp232 [ 1706.135972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1706.169354] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1706.990520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1707.027763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1707.054624] LNet: Removed LNI 192.168.203.53@tcp232 [ 1707.070403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1713.942660] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:40:58 (1788838858) [ 1714.970808] Key type .llcrypt unregistered [ 1714.980129] Key type ._llcrypt unregistered [ 1720.394344] Key type ._llcrypt registered [ 1720.399563] Key type .llcrypt registered [ 1720.598641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1720.610473] alg: No test for adler32 (adler32-zlib) [ 1721.546542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1721.568566] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1728.718865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1737.568567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.617962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1737.651468] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1738.958967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_interrupt -r 2 [ 1738.987995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 2 [ 1739.020553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_interrupt -r 2 [ 1739.053446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 2 [ 1739.098891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_interrupt -r 2 [ 1739.127824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 2 [ 1739.186836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_interrupt -r 2 [ 1739.224275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 2 [ 1739.260835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1739.279878] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1739.288967] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1739.551901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1739.582640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1739.606312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1739.644571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1739.678271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1739.710408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1739.749638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1739.787869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1739.799897] LNet: 88651:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1739.814810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1739.848977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1739.891315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1739.931887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1739.972820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1740.011603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1740.582281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_dropped -r 2 [ 1740.622979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 2 [ 1740.672657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_dropped -r 2 [ 1740.725144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 2 [ 1740.762625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_dropped -r 2 [ 1740.793677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 2 [ 1740.852966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_dropped -r 2 [ 1740.893842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 2 [ 1740.963199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1740.998324] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1741.263626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1741.313072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1741.341408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1741.376964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1741.410919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1741.442593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1741.468353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1741.497981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1741.520939] LNet: 88771:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1741.566730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1741.607555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1741.648683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1741.688807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1741.729742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1741.767912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1742.348484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_aborted -r 2 [ 1742.378365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 2 [ 1742.414370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_aborted -r 2 [ 1742.452924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 2 [ 1742.487600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_aborted -r 2 [ 1742.533450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 2 [ 1742.567579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_aborted -r 2 [ 1742.614747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 2 [ 1742.645976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1742.670161] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1742.972707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1743.004522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1743.029369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1743.055505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1743.094033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1743.126928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1743.152931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1743.180910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1743.201397] LNet: 88891:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1743.219667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1743.245592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1743.280695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1743.312709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1743.351268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1743.387293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1744.028275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_no_route -r 2 [ 1744.110774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 2 [ 1744.168958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_no_route -r 2 [ 1744.205374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 2 [ 1744.290510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_no_route -r 2 [ 1744.349357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 2 [ 1744.401180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_no_route -r 2 [ 1744.432608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 2 [ 1744.497328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1744.791486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1744.840130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1744.872150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1744.908946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1744.959516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1744.999562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1745.039713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1745.086415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1745.125701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1745.159995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1745.201508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1745.243738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1745.282995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1745.318921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1745.829662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_timeout -r 2 [ 1745.870476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 2 [ 1745.923954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_timeout -r 2 [ 1745.991961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 2 [ 1746.043459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_timeout -r 2 [ 1746.096564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 2 [ 1746.131951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_timeout -r 2 [ 1746.164530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 2 [ 1746.196100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1746.221904] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1746.227960] LNet: Skipped 1 previous similar message [ 1746.456544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1746.497037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1746.524333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1746.562951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1746.610213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1746.632327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1746.653682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1746.685242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1746.703338] LNet: 89132:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.712128] LNet: 89132:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1746.739719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1746.793908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1746.830980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1746.860225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1746.915356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1746.962813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1747.626766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_dropped -r 2 [ 1747.674387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 2 [ 1747.716345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_dropped -r 2 [ 1747.752531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 2 [ 1747.782982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_dropped -r 2 [ 1747.827401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 2 [ 1747.881045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_dropped -r 2 [ 1747.951408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 2 [ 1747.995889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1748.028842] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 1748.220171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1748.261410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1748.295790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1748.349462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1748.403032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1748.446125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1748.480073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1748.505105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1748.536301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1748.588039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1748.632156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1748.680718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1748.725335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1748.775035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1754.245935] LNet: 89326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.254376] LNetError: 89326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1754.268138] LNet: Removed LNI 192.168.203.53@tcp [ 1754.280844] LNet: Removed LNI 192.168.203.53@tcp1 [ 1754.696845] Key type .llcrypt unregistered [ 1754.698705] Key type ._llcrypt unregistered [ 1771.742191] Key type ._llcrypt registered [ 1771.744469] Key type .llcrypt registered [ 1773.175498] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:41:57 (1788838917) [ 1773.629708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.646943] alg: No test for adler32 (adler32-zlib) [ 1774.478816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.521426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1774.554276] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1775.304215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1775.343884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.53@tcp [ 1775.366196] LNetError: 90301:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1775.503619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.53@tcp [ 1775.653316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1782.820838] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:42:07 (1788838927) [ 1783.097075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.123340] LNet: 90892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.129886] LNetError: 90892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.146026] LNet: Removed LNI 192.168.203.53@tcp [ 1783.189713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.246564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.263246] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1783.988033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1784.024976] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1784.989292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.53@tcp [ 1785.011563] LNetError: 91158:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1785.311416] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1785.314056] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1786.384426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.53@tcp [ 1795.949869] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:42:20 (1788838940) [ 1798.534986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg353-server [ 1800.721291] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 23:42:25 (1788838945) [ 1802.754558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg353-server [ 1804.630860] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 23:42:29 (1788838949) [ 1806.316238] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg353-server [ 1808.181626] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:42:33 (1788838953) [ 1808.443950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.500789] LNet: 92780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.521083] LNetError: 92780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.532445] LNet: Removed LNI 192.168.203.53@tcp [ 1808.559776] LNet: Removed LNI 192.168.203.53@tcp1 [ 1808.585569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.681790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.737563] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1809.621089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1809.727386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1809.793803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1809.943911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1809.992379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1810.076830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1817.103145] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:42:41 (1788838961) [ 1817.194330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.242127] LNet: 93504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1817.250977] LNetError: 93504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1817.255870] LNetError: 93504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1817.290775] LNet: Removed LNI 192.168.203.53@tcp [ 1817.345850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1817.437345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1817.483456] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1825.948643] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:42:50 (1788838970) [ 1826.386632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.419178] LNet: 94215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.430894] LNetError: 94215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.468445] LNet: Removed LNI 192.168.203.53@tcp [ 1826.491083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.582206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1826.618734] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1833.022177] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:42:57 (1788838977) [ 1834.584461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1836.184323] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:43:01 (1788838981) [ 1836.774192] LNet: 95249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.788361] LNetError: 95249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.815805] LNet: Removed LNI 192.168.203.53@tcp [ 1837.436427] Key type .llcrypt unregistered [ 1837.441124] Key type ._llcrypt unregistered [ 1843.377761] Key type ._llcrypt registered [ 1843.380435] Key type .llcrypt registered [ 1843.573274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1843.585913] alg: No test for adler32 (adler32-zlib) [ 1844.563565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1844.613488] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1852.077168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1856.749741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.153@tcp [ 1857.126793] LNet: 95664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.137872] LNetError: 95664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.165503] LNet: Removed LNI 192.168.203.53@tcp [ 1857.659496] Key type .llcrypt unregistered [ 1857.663222] Key type ._llcrypt unregistered [ 1869.660824] Key type ._llcrypt registered [ 1869.663507] Key type .llcrypt registered [ 1871.011786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1873.145394] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1875.148218] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:43:39 (1788839019) [ 1877.495659] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1879.965317] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:43:44 (1788839024) [ 1882.061285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg353-server [ 1884.829946] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:43:49 (1788839029) [ 1888.145815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg353-server [ 1890.610046] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:43:54 (1788839034) [ 1892.677471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1892.707487] alg: No test for adler32 (adler32-zlib) [ 1893.708835] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 1893.924760] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1895.639167] Key type lgssc registered [ 1896.860158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1917.099331] Key type lgssc unregistered [ 1917.359728] LNet: 98836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.365582] LNetError: 98836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.381698] LNet: Removed LNI 192.168.203.53@tcp [ 1918.183379] Key type .llcrypt unregistered [ 1918.185068] Key type ._llcrypt unregistered [ 1935.611777] Key type ._llcrypt registered [ 1935.616894] Key type .llcrypt registered [ 1937.246456] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:44:41 (1788839081) [ 1937.790529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.820241] alg: No test for adler32 (adler32-zlib) [ 1938.783176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.859896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1938.900582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1938.961677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1939.086403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1947.373959] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:44:51 (1788839091) [ 1947.507485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.564144] LNet: 100244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.613593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.719527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.765902] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1948.899238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.939865] LNet: 100394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.952051] LNetError: 100394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.984278] LNet: Removed LNI 192.168.203.53@tcp [ 1949.019905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.114897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1949.161298] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1949.941579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.969368] LNet: 100422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.977750] LNetError: 100422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.994176] LNet: Removed LNI 192.168.203.53@tcp [ 1950.008624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.046744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1950.097921] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1958.144619] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:45:02 (1788839102) [ 1958.266250] LNet: 101005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.277633] LNetError: 101005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.305142] LNet: Removed LNI 192.168.203.53@tcp [ 1958.972834] Key type .llcrypt unregistered [ 1958.975680] Key type ._llcrypt unregistered [ 1965.955950] Key type ._llcrypt registered [ 1965.958487] Key type .llcrypt registered [ 1966.242729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1966.274848] alg: No test for adler32 (adler32-zlib) [ 1967.203849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1967.292196] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1967.295472] LNet: *** cfs_fail_loc=e003, val=100*** [ 1968.281167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1968.328542] LNet: *** cfs_fail_loc=e003, val=100*** [ 1968.345081] LNet: Skipped 2 previous similar messages [ 1968.361261] Ping reply received with mapping: [ 1968.363052] found metadata reply NI[192.168.203.53@tcp] [ 1968.368657] 35 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1968.374886] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1968.378002] 5a 5a 5a 5a 5a 5a 5a 5a [ 1968.554463] LNet: 101407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.560582] LNetError: 101407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.573218] LNet: Removed LNI 192.168.203.53@tcp [ 1969.333482] Key type .llcrypt unregistered [ 1969.336444] Key type ._llcrypt unregistered [ 1981.452873] Key type ._llcrypt registered [ 1981.455307] Key type .llcrypt registered [ 1982.853702] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:45:27 (1788839127) [ 1983.485387] Key type .llcrypt unregistered [ 1983.493160] Key type ._llcrypt unregistered [ 1989.951676] Key type ._llcrypt registered [ 1989.957826] Key type .llcrypt registered [ 1990.200281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1990.229265] alg: No test for adler32 (adler32-zlib) [ 1991.478400] LNetError: 102443:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1991.909033] Key type .llcrypt unregistered [ 1991.911254] Key type ._llcrypt unregistered [ 1997.300958] Key type ._llcrypt registered [ 1997.307079] Key type .llcrypt registered [ 1998.435838] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:45:43 (1788839143) [ 1998.932572] Key type .llcrypt unregistered [ 1998.934410] Key type ._llcrypt unregistered [ 2004.561331] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2004.569915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2007.188667] Key type ._llcrypt registered [ 2007.192379] Key type .llcrypt registered [ 2007.472624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2007.502498] alg: No test for adler32 (adler32-zlib) [ 2008.567914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.684104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2019.855598] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:46:04 (1788839164) [ 2019.993237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2020.050665] LNet: 104213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2020.064886] LNetError: 104213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2020.104128] LNet: Removed LNI 10.1.2.3@tcp [ 2020.144896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.844696] Key type .llcrypt unregistered [ 2020.848824] Key type ._llcrypt unregistered [ 2027.493815] Key type ._llcrypt registered [ 2027.498857] Key type .llcrypt registered [ 2027.682548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2027.705951] alg: No test for adler32 (adler32-zlib) [ 2028.644751] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2028.716856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2028.889568] LNet: 104504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.895521] LNetError: 104504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.913477] LNet: Removed LNI 192.168.203.53@tcp [ 2029.696560] Key type .llcrypt unregistered [ 2029.699468] Key type ._llcrypt unregistered [ 2040.878967] Key type ._llcrypt registered [ 2040.881351] Key type .llcrypt registered [ 2042.090909] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:46:27 (1788839187) [ 2042.748802] Key type .llcrypt unregistered [ 2042.752867] Key type ._llcrypt unregistered [ 2049.653501] Key type ._llcrypt registered [ 2049.655853] Key type .llcrypt registered [ 2049.772389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2049.803902] alg: No test for adler32 (adler32-zlib) [ 2058.136053] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:46:42 (1788839202) [ 2058.445074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.502279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.589118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.626072] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2059.583559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.642842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2059.695486] LNet: Removed LNI 192.168.203.53@tcp [ 2060.057524] LNet: 106244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.495375] Key type .llcrypt unregistered [ 2060.498392] Key type ._llcrypt unregistered [ 2075.134351] Key type ._llcrypt registered [ 2075.145715] Key type .llcrypt registered [ 2076.970250] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:47:01 (1788839221) [ 2077.464829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2077.488805] alg: No test for adler32 (adler32-zlib) [ 2078.469450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.603641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2078.653391] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2079.631722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2079.698889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.53@tcp [ 2079.741690] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2079.748404] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2079.759460] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2079, 0, 0 [ 2079.779545] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2079.787805] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2079.795863] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2079.802345] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2079, 0, 0 [ 2079.819068] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2079.824517] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2079.828581] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2079.832548] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2079, 0, 2129 [ 2079.847178] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2079.850268] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2079.853445] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2079.856527] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2079, 0, 2129 [ 2079.860724] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2086.322689] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:47:11 (1788839231) [ 2086.765235] LNet: 107782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2086.771516] LNetError: 107782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2086.788673] LNet: Removed LNI 192.168.203.53@tcp [ 2087.529434] Key type .llcrypt unregistered [ 2087.532639] Key type ._llcrypt unregistered [ 2093.890920] Key type ._llcrypt registered [ 2093.895589] Key type .llcrypt registered [ 2094.199909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2094.225225] alg: No test for adler32 (adler32-zlib) [ 2095.168292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2095.204485] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2103.712835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 2112.914939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.950491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2112.983044] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 2113.685985] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2113.748623] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2113.751966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2113.760635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2116.192260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2116.219361] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2127.803685] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:47:52 (1788839272) [ 2127.961392] LNet: 109280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.976833] LNetError: 109280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.987719] LNet: Removed LNI 192.168.203.53@tcp [ 2129.055532] LNet: Removed LNI 192.168.203.53@tcp1 [ 2129.061204] LNetError: 109280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2129.071448] LNetError: 109280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2129.101167] LNet: Removed LNI 10.1.2.3@tcp99 [ 2129.917710] Key type .llcrypt unregistered [ 2129.922694] Key type ._llcrypt unregistered [ 2136.487357] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2136.497775] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2139.131567] Key type ._llcrypt registered [ 2139.136930] Key type .llcrypt registered [ 2139.351109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2139.375101] alg: No test for adler32 (adler32-zlib) [ 2140.368823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.429489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.459665] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2141.234089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2141.261751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2143.555598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2143.616211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2143.651278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2143.671138] LNetError: 110026:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2143.716804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2143.770895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2144.224159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2144.336910] LNet: 110046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.343371] LNetError: 110046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2144.349736] LNet: Removed LNI 10.1.2.3@tcp [ 2145.381899] LNet: Removed LNI 192.168.203.53@tcp [ 2146.120430] Key type .llcrypt unregistered [ 2146.124448] Key type ._llcrypt unregistered [ 2159.201703] Key type ._llcrypt registered [ 2159.203417] Key type .llcrypt registered [ 2160.685288] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:48:25 (1788839305) [ 2161.213444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2161.224166] alg: No test for adler32 (adler32-zlib) [ 2162.197625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.255157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2162.280985] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2163.147056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2163.264814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg353-client@tcp [ 2169.957314] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:48:34 (1788839314) [ 2170.297195] LNet: 111576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2170.314903] LNetError: 111576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2171.370426] LNet: Removed LNI 192.168.203.53@tcp [ 2172.031544] Key type .llcrypt unregistered [ 2172.037531] Key type ._llcrypt unregistered [ 2178.421239] Key type ._llcrypt registered [ 2178.427257] Key type .llcrypt registered [ 2178.621921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.648536] alg: No test for adler32 (adler32-zlib) [ 2179.572663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2179.609871] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2179.633041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2186.447889] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:48:51 (1788839331) [ 2187.918485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2189.674680] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:48:54 (1788839334) [ 2191.340954] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2193.199623] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:48:57 (1788839337) [ 2193.358410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.383432] LNet: 112994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.395280] LNetError: 112994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.431315] LNet: Removed LNI 192.168.203.53@tcp [ 2193.465576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.520784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.541193] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2194.434370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2194.463691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2201.892810] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:49:06 (1788839346) [ 2202.098342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.157888] LNet: 113706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.170855] LNetError: 113706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.204883] LNet: Removed LNI 192.168.203.53@tcp [ 2202.232162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.300024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2202.364283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2203.081766] Key type .llcrypt unregistered [ 2203.088691] Key type ._llcrypt unregistered [ 2216.926781] Key type ._llcrypt registered [ 2216.930534] Key type .llcrypt registered [ 2218.751467] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:49:22 (1788839362) [ 2219.090765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2219.114373] alg: No test for adler32 (adler32-zlib) [ 2220.032355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2220.104816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2220.154780] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2220.814530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2220.868961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2227.745648] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:49:32 (1788839372) [ 2227.895697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.933878] LNet: 115256:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2227.958147] LNetError: 115256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2228.003109] LNet: Removed LNI 192.168.203.53@tcp [ 2228.051433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.163987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2228.237407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2236.087719] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:49:40 (1788839380) [ 2236.285183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2236.313254] LNet: 115842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2236.350345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2236.466305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2236.940758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2236.988453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2237.053273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2237.360692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2237.398728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2237.484295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.515980] LNet: 115875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.539693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2237.597568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2238.017894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2238.080085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2238.142657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2238.341915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2238.373817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2245.186150] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:49:49 (1788839389) [ 2245.314542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2245.347724] LNet: 116475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2245.389636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2245.458936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2245.517821] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2246.382812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2246.402620] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2247.323490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2247.432419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2247.722119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.744611] LNetError: 116755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.800835] LNet: Removed LNI 192.168.203.53@tcp [ 2248.803566] LNetError: 116755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.817811] LNet: Removed LNI 192.168.203.53@tcp2 [ 2248.855468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.934617] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2248.953461] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2248.968589] LNet: 116761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2249.549398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2249.709409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2249.731872] LNet: 116798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.737587] LNet: 116798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2250.783151] LNet: Removed LNI 192.168.203.53@tcp [ 2250.795718] LNet: Removed LNI 192.168.203.53@tcp2 [ 2250.845119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.920763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2250.952272] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2251.570893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2251.612101] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2252.514373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2252.566349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2252.650980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2252.909493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.945703] LNetError: 117083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2252.958813] LNetError: 117083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2252.965558] LNet: Removed LNI 192.168.203.53@tcp [ 2253.991416] LNet: Removed LNI 192.168.203.53@tcp2 [ 2254.045813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.134977] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2254.143529] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2254.149804] LNet: 117089:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2254.641715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2254.848789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.900485] LNet: Removed LNI 192.168.203.53@tcp [ 2255.975511] LNet: Removed LNI 192.168.203.53@tcp2 [ 2255.997287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2256.053510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2256.106399] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2256.856679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2256.879571] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2257.757688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2257.827963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2258.256068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.296789] LNet: 117406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.302988] LNet: 117406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2258.310940] LNetError: 117406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.316878] LNetError: 117406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2258.326332] LNet: Removed LNI 192.168.203.53@tcp [ 2258.340520] LNet: Removed LNI 192.168.203.53@tcp2 [ 2258.379661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.490412] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2258.504351] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2258.511172] LNet: 117412:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2259.132975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2259.356731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.447768] LNet: Removed LNI 192.168.203.53@tcp [ 2260.457682] LNet: Removed LNI 192.168.203.53@tcp2 [ 2260.486368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.548793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.578416] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2261.331507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2261.351411] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2262.104912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2262.150970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2262.224718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2262.398277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2263.455140] LNet: Removed LNI 192.168.203.53@tcp [ 2264.511532] LNet: Removed LNI 192.168.203.53@tcp2 [ 2264.588954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.739748] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2264.752977] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2264.759812] LNet: 117740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2265.250278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2265.553288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.598342] LNet: Removed LNI 192.168.203.53@tcp [ 2266.669159] LNet: Removed LNI 192.168.203.53@tcp2 [ 2266.718571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.840565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2266.889672] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2267.494300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2267.510512] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2268.193887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2268.214926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2268.253387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2268.634356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2268.659996] LNetError: 118073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.677270] LNetError: 118073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2268.690852] LNet: Removed LNI 192.168.203.53@tcp [ 2268.702758] LNet: Removed LNI 192.168.203.53@tcp2 [ 2268.742570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.868374] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2268.886424] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2268.907216] LNet: 118079:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.324037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2269.667897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.751705] LNet: Removed LNI 192.168.203.53@tcp [ 2270.769856] LNet: Removed LNI 192.168.203.53@tcp2 [ 2270.860823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.965492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.028137] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2271.936293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2271.957796] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2273.211887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2273.309053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2273.381959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2273.474349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2274.016536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2275.103224] LNet: Removed LNI 192.168.203.53@tcp [ 2276.136770] LNet: Removed LNI 192.168.203.53@tcp2 [ 2276.177988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2276.323876] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2276.336579] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2277.296964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2286.541921] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2288.842250] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:50:33 (1788839433) [ 2289.150713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2289.187490] LNet: 119200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.192753] LNet: 119200:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 2289.204823] LNetError: 119200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2289.210378] LNetError: 119200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2289.216736] LNet: Removed LNI 192.168.203.53@tcp [ 2290.280802] LNet: Removed LNI 192.168.203.53@tcp2 [ 2290.339735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.751778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.785147] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2291.749471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2291.784225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2293.659865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.713829] LNet: Removed LNI 10.1.2.3@tcp [ 2293.726859] LNet: Removed LNI 192.168.203.53@tcp [ 2293.778753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.891809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2293.927288] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2293.932741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2294.429171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2294.467090] LNet: Removed LNI 192.168.203.53@tcp [ 2294.483853] LNet: Removed LNI 10.1.2.3@tcp [ 2301.465710] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 23:50:46 (1788839446) [ 2301.546586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.594338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.655993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.747663] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2302.631900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2302.664350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.690252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.720403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.746263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.769870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.802315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.838200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.872330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.905769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2302.940417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2303.044740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2309.863067] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:50:54 (1788839454) [ 2309.965944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.079622] LNet: Removed LNI 192.168.203.53@tcp [ 2311.122899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.291529] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2311.370521] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2311.376949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2311.390535] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2313.885940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2313.923631] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2315.071783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2315.101077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2317.593263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2317.627677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2317.691173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.53@tcp -m PUT -e local_timeout -r 1 [ 2317.769586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2317.870831] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2317.879545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2318.128528] LNetError: 120883:0:(lib-move.c:4801:lnet_parse()) 192.168.203.53@tcp, src 192.168.203.53@tcp: Dropping PUT (error -108 looking up sender) [ 2320.223171] LNet: Removed LNI 10.1.2.3@tcp [ 2321.257978] LNet: Removed LNI 192.168.203.53@tcp [ 2321.660360] Key type .llcrypt unregistered [ 2321.664464] Key type ._llcrypt unregistered [ 2333.188606] Key type ._llcrypt registered [ 2333.190830] Key type .llcrypt registered [ 2334.704209] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:51:19 (1788839479) [ 2335.179034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2335.220956] alg: No test for adler32 (adler32-zlib) [ 2336.122861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2336.620620] LNet: 122191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.597316] Key type .llcrypt unregistered [ 2337.599164] Key type ._llcrypt unregistered [ 2346.685331] Key type ._llcrypt registered [ 2346.690170] Key type .llcrypt registered [ 2347.120621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2347.176423] alg: No test for adler32 (adler32-zlib) [ 2348.265038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2348.323985] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2358.408049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 2358.865230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2360.052618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.153@tcp [ 2360.620274] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 2360.725881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2360.821603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2360.871969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2360.953729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2361.020915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2361.081987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2361.130865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2361.203264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2361.237384] LNet: 122639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.304770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2361.373589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2361.439491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2361.523912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2361.575767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2361.646122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2362.140423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2362.321465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2362.362876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2362.442690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2362.481778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2362.508795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2362.534859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2362.607679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2362.686462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2362.712810] LNet: 122757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2362.732362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2362.787459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2362.856225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2362.905601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2362.968885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2363.015309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2363.642404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.153@tcp [ 2363.923591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2363.961253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2364.010238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2364.045426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2364.072994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2364.103821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2364.134748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2364.170885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.188538] LNet: 122874:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.206903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2364.230472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2364.268524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2364.293089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2364.326635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2364.364525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.082575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2365.348465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2365.371599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2365.393685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2365.418952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2365.440824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2365.465334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2365.495763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2365.540848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2365.567594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2365.592571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2365.618269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2365.654692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2365.674703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2365.702369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.995562] LNet: 123037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2366.001747] LNetError: 123037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2366.021741] LNet: Removed LNI 192.168.203.53@tcp [ 2366.524828] Key type .llcrypt unregistered [ 2366.528349] Key type ._llcrypt unregistered [ 2384.203950] Key type ._llcrypt registered [ 2384.207246] Key type .llcrypt registered [ 2385.386910] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:52:10 (1788839530) [ 2385.774757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2385.814814] alg: No test for adler32 (adler32-zlib) [ 2386.789930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.284581] LNet: 123923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2388.054907] Key type .llcrypt unregistered [ 2388.057579] Key type ._llcrypt unregistered [ 2395.222603] Key type ._llcrypt registered [ 2395.225803] Key type .llcrypt registered [ 2395.449893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2395.468424] alg: No test for adler32 (adler32-zlib) [ 2396.388085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2396.410674] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2404.740822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 2413.743905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.798087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2413.833856] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 2415.115434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2415.854670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.153@tcp [ 2415.907192] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 2416.114510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2416.171542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2416.248608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2416.313732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2416.388132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2416.455309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2416.518204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2416.558269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2416.597459] LNet: 124573:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2416.629823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2416.683211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2416.722927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2416.763313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2416.831395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2416.884625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2417.665980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2417.981892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.024949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2418.070299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.108985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.134033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.160416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.201713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.258388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.289703] LNet: 124691:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.317143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.346309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.376566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.399592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.423614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.444499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2419.071179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.153@tcp [ 2419.382728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2419.429857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2419.476452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2419.516876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2419.564408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2419.599626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2419.634670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2419.670307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2419.688572] LNet: 124808:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2419.701433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2419.723901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2419.749915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2419.773455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2419.793395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2419.821277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2420.476185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2420.692501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2420.721191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2420.752953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.775428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.797674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.818992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.844683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.867851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.892517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.918365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.941405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.997790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2421.050707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2421.108569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2425.961913] LNet: 125006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2425.982134] LNetError: 125006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2425.987688] LNet: Removed LNI 192.168.203.53@tcp [ 2425.999979] LNet: Removed LNI 192.168.203.53@tcp1 [ 2426.432760] Key type .llcrypt unregistered [ 2426.437496] Key type ._llcrypt unregistered [ 2442.135206] Key type ._llcrypt registered [ 2442.138634] Key type .llcrypt registered [ 2443.779906] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 23:53:08 (1788839588) [ 2444.169942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.190983] alg: No test for adler32 (adler32-zlib) [ 2445.156354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.201577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.224053] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2445.907466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2445.936617] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2446.833737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2446.939668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2447.967221] LNet: Removed LNI 192.168.203.53@tcp [ 2454.342416] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 23:53:19 (1788839599) [ 2455.598322] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2456.979634] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:53:22 (1788839602) [ 2465.935348] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:53:30 (1788839610) [ 2475.699532] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:53:40 (1788839620) [ 2485.405128] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:53:50 (1788839630) [ 2491.932513] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:53:57 (1788839637) [ 2500.043543] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:54:04 (1788839644) [ 2508.229948] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:54:13 (1788839653) [ 2515.462726] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:54:20 (1788839660) [ 2522.344821] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:54:27 (1788839667) [ 2529.263492] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:54:33 (1788839673) [ 2536.501811] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:54:41 (1788839681) [ 2543.865485] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:54:48 (1788839688) [ 2551.411209] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:54:56 (1788839696) [ 2558.773981] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:55:03 (1788839703) [ 2566.142193] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:55:10 (1788839710) [ 2574.943586] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:55:19 (1788839719) [ 2582.475354] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:55:27 (1788839727) [ 2591.193463] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:55:35 (1788839735) [ 2591.445918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2591.486522] LNet: 136698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2591.499799] LNetError: 136698:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2591.514743] LNet: Removed LNI 192.168.203.53@tcp2 [ 2591.531450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2591.689987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2591.716922] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2593.025506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2593.055943] LNet: 136840:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2593.064274] LNetError: 136840:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2593.079878] LNet: Removed LNI 192.168.203.53@tcp [ 2593.101523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2593.177611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2593.207056] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2600.350797] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:55:44 (1788839744) [ 2606.972249] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:55:52 (1788839752) [ 2614.656857] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:55:59 (1788839759) [ 2622.571565] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:56:07 (1788839767) [ 2631.013206] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:56:15 (1788839775) [ 2639.951303] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:56:24 (1788839784) [ 2648.237125] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:56:32 (1788839792) [ 2656.186284] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:56:41 (1788839801) [ 2663.436688] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:56:48 (1788839808) [ 2671.852195] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:56:56 (1788839816) [ 2679.305537] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:57:04 (1788839824) [ 2686.945278] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:57:11 (1788839831) [ 2694.423525] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2451 sec ========== 23:57:19 (1788839839) [ 2694.747185] LNet: 144307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2694.752639] LNetError: 144307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2694.764289] LNet: Removed LNI 192.168.203.53@tcp [ 2695.506151] Key type .llcrypt unregistered [ 2695.508438] Key type ._llcrypt unregistered [ 2709.774552] Key type ._llcrypt registered [ 2709.787618] Key type .llcrypt registered [ 2710.806759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2710.832370] alg: No test for adler32 (adler32-zlib) [ 2711.818700] Lustre: Lustre: Build Version: 2.17.57_113_gf5ca795 [ 2712.015936] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2713.663512] Key type lgssc registered [ 2715.386697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2785.608190] Lustre: Mounted lustre-client - version 2.17.57_113_gf5ca795 [ 2791.430861] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2800.380569] Lustre: Unmounted lustre-client [ 2824.156373] Key type lgssc unregistered [ 2824.388977] LNet: 147257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2824.399301] LNetError: 147257:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2824.413921] LNet: Removed LNI 192.168.203.53@tcp [ 2825.122470] Key type .llcrypt unregistered [ 2825.125144] Key type ._llcrypt unregistered