[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481600237 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003228] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.006017] kvm-guest: setup PV IPIs [ 0.009866] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012007] pid_max: default: 32768 minimum: 301 [ 0.013159] LSM: Security Framework initializing [ 0.014060] Yama: becoming mindful. [ 0.015033] SELinux: Initializing. [ 0.016063] *** VALIDATE selinux *** [ 0.024773] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029484] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033094] *** VALIDATE tmpfs *** [ 0.034458] *** VALIDATE proc *** [ 0.035221] *** VALIDATE cgroup *** [ 0.036009] *** VALIDATE cgroup2 *** [ 0.037262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044215] debug: unmapping init [mem 0xffffffffb2659000-0xffffffffb2660fff] [ 0.047000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047688] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048026] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052011] ... max period: 00007fffffffffff [ 0.053011] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055319] rcu: Hierarchical SRCU implementation. [ 0.057506] smp: Bringing up secondary CPUs ... [ 0.058599] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.069163] smp: Brought up 1 node, 4 CPUs [ 0.071013] smpboot: Max logical packages: 1 [ 0.072013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.190000] node 0 deferred pages initialised in 115ms [ 0.192129] devtmpfs: initialized [ 0.193265] x86/mm: Memory block size: 128MB [ 0.196909] gcov: version magic: 0x41383552 [ 0.199033] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.202120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.205357] pinctrl core: initialized pinctrl subsystem [ 0.207179] [ 0.207800] ************************************************************* [ 0.210012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.212008] ** ** [ 0.214011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.216105] ** ** [ 0.219010] ** This means that this kernel is built to expose internal ** [ 0.221009] ** IOMMU data structures, which may compromise security on ** [ 0.224012] ** your system. ** [ 0.226008] ** ** [ 0.228014] ** If you see this message and you are not debugging the ** [ 0.230011] ** kernel, report this immediately to your vendor! ** [ 0.232019] ** ** [ 0.235011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.237015] ************************************************************* [ 0.240301] NET: Registered protocol family 16 [ 0.241562] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.244059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.247060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.251045] cpuidle: using governor menu [ 0.252791] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.255411] PCI: Using configuration type 1 for base access [ 0.257123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.267142] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.270060] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.272128] cryptd: max_cpu_qlen set to 1000 [ 0.275352] ACPI: Added _OSI(Module Device) [ 0.276000] ACPI: Added _OSI(Processor Device) [ 0.276000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.277027] ACPI: Added _OSI(Processor Aggregator Device) [ 0.284222] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.290569] ACPI: Interpreter enabled [ 0.293086] ACPI: PM: (supports S0 S3 S4 S5) [ 0.296016] ACPI: Using IOAPIC for interrupt routing [ 0.299110] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.305924] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.320000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.325045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.329021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.333079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.339844] acpiphp: Slot [2] registered [ 0.341116] acpiphp: Slot [5] registered [ 0.343104] acpiphp: Slot [6] registered [ 0.344091] acpiphp: Slot [3] registered [ 0.346108] acpiphp: Slot [4] registered [ 0.347076] acpiphp: Slot [7] registered [ 0.348074] acpiphp: Slot [8] registered [ 0.350107] acpiphp: Slot [9] registered [ 0.351129] acpiphp: Slot [10] registered [ 0.353095] acpiphp: Slot [11] registered [ 0.355100] acpiphp: Slot [12] registered [ 0.356138] acpiphp: Slot [13] registered [ 0.358086] acpiphp: Slot [14] registered [ 0.359091] acpiphp: Slot [15] registered [ 0.361083] acpiphp: Slot [16] registered [ 0.362171] acpiphp: Slot [17] registered [ 0.364101] acpiphp: Slot [18] registered [ 0.366086] acpiphp: Slot [19] registered [ 0.367081] acpiphp: Slot [20] registered [ 0.369084] acpiphp: Slot [21] registered [ 0.370108] acpiphp: Slot [22] registered [ 0.372106] acpiphp: Slot [23] registered [ 0.373076] acpiphp: Slot [24] registered [ 0.375087] acpiphp: Slot [25] registered [ 0.376086] acpiphp: Slot [26] registered [ 0.378090] acpiphp: Slot [27] registered [ 0.379081] acpiphp: Slot [28] registered [ 0.381102] acpiphp: Slot [29] registered [ 0.382192] acpiphp: Slot [30] registered [ 0.384092] acpiphp: Slot [31] registered [ 0.385073] PCI host bridge to bus 0000:00 [ 0.387018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.389019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.391020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.394018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.397022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.400026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.402176] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.405130] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.408395] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.415988] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.420055] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.424019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.426015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.428016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.431620] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.434801] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.437043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.439833] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.443785] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.452017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.457012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.461608] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.468015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.474015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.488017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.497097] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.511018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.521019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.544016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.559347] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.563380] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.566420] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.569431] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.572241] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.579074] iommu: Default domain type: Passthrough [ 0.581496] SCSI subsystem initialized [ 0.583141] ACPI: bus type USB registered [ 0.584101] usbcore: registered new interface driver usbfs [ 0.586103] usbcore: registered new interface driver hub [ 0.589096] usbcore: registered new device driver usb [ 0.592146] pps_core: LinuxPPS API ver. 1 registered [ 0.594012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.597047] PTP clock support registered [ 0.599033] EDAC MC: Ver: 3.0.0 [ 0.600129] PCI: Using ACPI for IRQ routing [ 0.601361] NetLabel: Initializing [ 0.603012] NetLabel: domain hash size = 128 [ 0.604007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.605063] NetLabel: unlabeled traffic allowed by default [ 0.607137] vgaarb: loaded [ 0.608265] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.609011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.615580] clocksource: Switched to clocksource kvm-clock [ 0.743477] VFS: Disk quotas dquot_6.6.0 [ 0.745657] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.749734] *** VALIDATE ramfs *** [ 0.751504] *** VALIDATE hugetlbfs *** [ 0.752994] pnp: PnP ACPI init [ 0.754741] pnp: PnP ACPI: found 6 devices [ 0.770483] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.773896] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.776270] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.778614] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.780832] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.783352] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.786272] NET: Registered protocol family 2 [ 0.788713] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.793977] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.797567] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.802673] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.806092] TCP: Hash tables configured (established 65536 bind 65536) [ 0.808980] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.812078] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.814837] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.817768] NET: Registered protocol family 1 [ 0.821311] RPC: Registered named UNIX socket transport module. [ 0.823605] RPC: Registered udp transport module. [ 0.825536] RPC: Registered tcp transport module. [ 0.827269] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.829716] NET: Registered protocol family 44 [ 0.831453] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.833688] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.835878] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.838425] PCI: CLS 0 bytes, default 64 [ 0.840175] Unpacking initramfs... [ 2.654402] debug: unmapping init [mem 0xffff9af47cc64000-0xffff9af47ffcffff] [ 2.658563] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.661022] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.663997] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.388716] Initialise system trusted keyrings [ 3.391981] Key type blacklist registered [ 3.394071] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.404484] zbud: loaded [ 3.409405] *** VALIDATE nfs *** [ 3.410765] *** VALIDATE nfs4 *** [ 3.412670] pstore: using deflate compression [ 3.416904] Platform Keyring initialized [ 3.579888] NET: Registered protocol family 38 [ 3.581668] Key type asymmetric registered [ 3.583201] Asymmetric key parser 'x509' registered [ 3.585294] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.588698] io scheduler mq-deadline registered [ 3.590109] io scheduler kyber registered [ 3.591983] io scheduler bfq registered [ 3.594174] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.596803] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.599688] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.602937] ACPI: Power Button [PWRF] [ 3.614967] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.623930] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.649553] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.683468] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.716725] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.722509] Non-volatile memory driver v1.3 [ 3.724162] Linux agpgart interface v0.103 [ 3.753331] virtio_blk virtio1: [vda] 136000 512-byte logical blocks (69.6 MB/66.4 MiB) [ 3.757104] vda: detected capacity change from 0 to 69632000 [ 3.770726] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.773662] vdb: detected capacity change from 0 to 1073741824 [ 3.780157] libphy: Fixed MDIO Bus: probed [ 3.792706] usbcore: registered new interface driver usbserial_generic [ 3.811767] usbserial: USB Serial support registered for generic [ 3.815656] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.823500] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.826222] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.851566] mousedev: PS/2 mouse device common for all mice [ 3.856648] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.857052] rtc_cmos 00:05: RTC can wake from S4 [ 3.866650] rtc_cmos 00:05: registered as rtc0 [ 3.869751] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.873046] intel_pstate: CPU model not supported [ 3.877785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.879441] hid: raw HID events driver (C) Jiri Kosina [ 3.889679] usbcore: registered new interface driver usbhid [ 3.892091] usbhid: USB HID core driver [ 3.895926] drop_monitor: Initializing network drop monitor service [ 3.898213] Initializing XFRM netlink socket [ 3.899780] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.902445] NET: Registered protocol family 10 [ 3.906966] Segment Routing with IPv6 [ 3.907801] NET: Registered protocol family 17 [ 3.909516] mpls_gso: MPLS GSO support [ 3.913144] RAS: Correctable Errors collector initialized. [ 3.914785] AVX version of gcm_enc/dec engaged. [ 3.916054] AES CTR mode by8 optimization enabled [ 3.987370] sched_clock: Marking stable (3987296244, 0)->(4880528237, -893231993) [ 3.992551] registered taskstats version 1 [ 3.994357] Loading compiled-in X.509 certificates [ 3.995936] zswap: loaded using pool lzo/zbud [ 4.026736] Key type big_key registered [ 4.040322] Key type encrypted registered [ 4.041529] ima: No TPM chip found, activating TPM-bypass! [ 4.043298] ima: Allocated hash algorithm: sha1 [ 4.044555] ima: No architecture policies found [ 4.045933] evm: Initialising EVM extended attributes: [ 4.047349] evm: security.selinux [ 4.048298] evm: security.ima [ 4.049194] evm: security.capability [ 4.050201] evm: HMAC attrs: 0x1 [ 4.052225] rtc_cmos 00:05: setting system clock to 2026-06-03 01:09:14 UTC (1780448954) [ 4.058782] debug: unmapping init [mem 0xffffffffb3603000-0xffffffffb37fffff] [ 4.061454] debug: unmapping init [mem 0xffffffffb2382000-0xffffffffb2658fff] [ 4.070227] Write protecting the kernel read-only data: 28672k [ 4.073971] debug: unmapping init [mem 0xffffffffb0a03000-0xffffffffb0bfffff] [ 4.076485] debug: unmapping init [mem 0xffffffffb1314000-0xffffffffb13fffff] [ 4.110620] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.119391] systemd[1]: Detected virtualization kvm. [ 4.121128] systemd[1]: Detected architecture x86-64. [ 4.122625] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.150783] systemd[1]: No hostname configured. [ 4.153565] systemd[1]: Set hostname to . [ 4.155636] random: systemd: uninitialized urandom read (16 bytes read) [ 4.158309] systemd[1]: Initializing machine ID from random generator. [ 4.232121] random: ln: uninitialized urandom read (6 bytes read) [ 4.337462] random: systemd: uninitialized urandom read (16 bytes read) [ 4.339925] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.346581] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.356428] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.112141] device-mapper: uevent: version 1.0.3 [ 5.114100] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.114636] virtio_net virtio0 ens2: renamed from eth0 [ 6.122927] random: fast init done [ 6.248285] scsi host0: ata_piix [ 6.272570] scsi host1: ata_piix [ 6.277422] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.283508] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 10.737225] random: crng init done [ 10.739414] random: 7 urandom warning(s) missed due to ratelimiting [ 10.974146] dracut-initqueue[584]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 12.418901] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 14.141761] printk: systemd: 26 output lines suppressed due to ratelimiting [ 14.438518] SELinux: Disabled at runtime. [ 14.499742] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.505867] systemd[1]: Detected virtualization kvm. [ 14.507559] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.115895] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.119669] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.130571] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.136424] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.140990] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.148550] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.164650] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on ude[ 15.406315] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS v Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 15.892898] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 16.523817] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.620387] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 16.867345] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.887599] EDAC sbridge: Ver: 1.1.2 [ 18.498202] Key type dns_resolver registered [ 18.805147] NFS: Registering the id_resolver key type [ 18.807135] Key type id_resolver registered [ 18.808742] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg112-client login: [ 42.147333] hrtimer: interrupt took 25657277 ns [ 92.871891] libcfs: loading out-of-tree module taints kernel. [ 93.137214] Key type ._llcrypt registered [ 93.140932] Key type .llcrypt registered [ 93.784806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 93.821973] alg: No test for adler32 (adler32-zlib) [ 95.681431] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 97.273196] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 99.352109] Key type lgssc registered [ 102.022922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 251.501218] Lustre: Mounted lustre-client [ 257.641095] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 271.355052] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing check_logdir /tmp/testlogs/ [ 276.959208] Lustre: lustre-OST0000-osc-ffff9af4cb887000: disconnect after 23s idle [ 277.607301] Lustre: DEBUG MARKER: oleg112-client.virtnet: executing yml_node [ 282.417229] Lustre: DEBUG MARKER: Client: 2.17.53.57 [ 285.612491] Lustre: DEBUG MARKER: MDS: 2.17.53.57 [ 288.906343] Lustre: DEBUG MARKER: OSS: 2.17.53.57 [ 290.518434] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Jun 2 21:13:59 EDT 2026 [ 309.866669] Lustre: Unmounted lustre-client [ 343.535966] Key type lgssc unregistered [ 343.916809] LNet: 8378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.931220] LNetError: 8378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 343.947353] LNet: Removed LNI 192.168.201.12@tcp [ 344.854195] Key type .llcrypt unregistered [ 344.858536] Key type ._llcrypt unregistered [ 378.684748] Key type ._llcrypt registered [ 378.689519] Key type .llcrypt registered [ 378.824192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 378.843990] alg: No test for adler32 (adler32-zlib) [ 380.067098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 380.153477] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 380.170455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 385.569728] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 385.886769] LNet: 9265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.891132] LNetError: 9265:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 385.905719] LNet: Removed LNI 192.168.201.12@tcp [ 386.637291] Key type .llcrypt unregistered [ 386.639314] Key type ._llcrypt unregistered [ 394.697808] Key type ._llcrypt registered [ 394.703712] Key type .llcrypt registered [ 396.233493] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 21:15:44 (1780449344) [ 396.819527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 396.845323] alg: No test for adler32 (adler32-zlib) [ 397.805481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.874953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.891736] LNet: 9677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.912823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.002382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 398.032584] LNet: 9684:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 405.812204] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 21:15:54 (1780449354) [ 406.114133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.156578] LNet: 10261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.201390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.324439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 413.937974] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 21:16:02 (1780449362) [ 414.150578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.175233] LNet: 10847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.190880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.285751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 422.304575] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 21:16:10 (1780449370) [ 422.465977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.521256] LNet: 11425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.584745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.798330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 431.669180] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 21:16:20 (1780449380) [ 431.813866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.857180] LNet: 12011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.915716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.085405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 432.864867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 440.839833] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 21:16:29 (1780449389) [ 441.275662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.311533] LNet: 12610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 441.353621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.454254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 442.103888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 442.952480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 451.301770] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 21:16:40 (1780449400) [ 451.431693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.510675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.687293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 460.035369] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 21:16:49 (1780449409) [ 460.426746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.473620] LNet: 13792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.480062] LNet: 13792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 460.510740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.606824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 460.634517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 461.051234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 461.083258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 461.501756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 461.536656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 462.064782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 462.109064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 462.505453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 462.553964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 463.013190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 463.054547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 463.570756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 463.602325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 464.148196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 464.185543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 464.652370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 464.695773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 464.988613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 465.035733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 465.549910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 465.625900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 466.169356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 466.226847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 466.642939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 466.703210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 474.787914] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:17:03 (1780449423) [ 475.102692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.157792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.269764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 475.312640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 483.918870] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:17:12 (1780449432) [ 484.081727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.124477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.188126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 484.214908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 491.652816] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:17:20 (1780449440) [ 491.846890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.909954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.069663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 492.138553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 500.074947] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:17:28 (1780449448) [ 500.207966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.268085] LNet: 16229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.296456] LNet: 16229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 500.362283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.410841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 500.443898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 510.137820] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:17:39 (1780449459) [ 510.409559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.483536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.654995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 510.716629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 519.285668] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:17:48 (1780449468) [ 519.489236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.565863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.719122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 519.783903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 529.628522] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:17:57 (1780449477) [ 529.727599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.798804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.962486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 530.747640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 539.830543] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:18:08 (1780449488) [ 539.996499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.058186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.331574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.647810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 551.526667] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:18:19 (1780449499) [ 551.677689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.750633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.054536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 552.256031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 561.519518] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:18:29 (1780449509) [ 561.705987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.756286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.427275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.744547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 570.702373] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:18:39 (1780449519) [ 570.922842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.967757] LNet: 20482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 570.972556] LNet: 20482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 571.035734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.153698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 571.226515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 580.881267] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:18:49 (1780449529) [ 581.008140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.097396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.274786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 581.371627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 588.987372] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:18:57 (1780449537) [ 589.294637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.324878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.385288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 596.605769] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:19:05 (1780449545) [ 596.869389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.912936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.996033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 604.495995] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:19:13 (1780449553) [ 604.846868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.951576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.091932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 605.938624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 613.675665] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:19:22 (1780449562) [ 613.954783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.000775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.109644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 614.149528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 622.484807] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:19:31 (1780449571) [ 622.701614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.730606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.824233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 622.864143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 630.176511] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:19:39 (1780449579) [ 630.507190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.555497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.624849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 630.659356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 638.286910] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:19:47 (1780449587) [ 638.635439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 638.689302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 638.800454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 638.865459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 647.198751] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:19:56 (1780449596) [ 647.600801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.657769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.748935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 647.923521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 647.998605] LNetError: 25799:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 648.128409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 648.193781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 656.682195] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:20:05 (1780449605) [ 656.821052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.929946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.119630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SKkpPu0Hfp [ 657.212350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SKkpPu0Hfp [ 657.268921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SKkpPu0Hfp [ 664.127989] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:20:13 (1780449613) [ 664.226579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.291950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.396886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 664.467841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 664.528650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 672.342757] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:20:20 (1780449620) [ 672.442472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.498684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.658812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 676.147932] LNet: Message forwarding will be enabled in 2s [ 679.263399] LNet: Message forwarding enabled [ 679.785307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 679.821710] LNet: Message forwarding will stop in 2s to 6s [ 684.671344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 685.175468] LNet: Message forwarding disabled [ 687.454178] LNet: Message forwarding will be enabled in 2s [ 691.368255] LNet: Message forwarding enabled [ 695.539573] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:20:44 (1780449644) [ 695.698296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.743524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.867391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 704.732470] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:20:53 (1780449653) [ 705.052500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.075808] LNet: 28829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.080521] LNet: 28829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 705.098705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.148361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 705.178823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 705.229322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 705.264370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 705.301329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 705.337754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 705.366845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 705.429824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 705.489517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 705.544919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 705.639394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 705.686227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 705.747830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 705.793334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 705.859983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 705.914920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 705.964415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 706.029889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 706.067757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 706.084444] LNetError: 28873:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 712.628386] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:21:01 (1780449661) [ 712.993924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.042557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.127205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 719.391479] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:21:08 (1780449668) [ 721.202391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg112-server [ 722.978611] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:21:12 (1780449672) [ 723.183442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.208516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.256184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 723.290908] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 724.559861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[13-17]@tcp [ 724.584748] LNet: 30590:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 724.650500] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 724.659850] LNetError: 30380:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.13@tcp has gone from up to down [ 725.608534] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 725.622571] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 726.631821] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 726.638768] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 728.687129] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.13@tcp is being used as a gateway but routing feature is not turned on [ 728.718280] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 731.207549] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:21:20 (1780449680) [ 732.770776] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 732.798520] LNetError: 30380:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 732.823700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg112-server [ 734.568514] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:21:23 (1780449683) [ 735.010717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.052013] LNetError: 31527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.074457] LNet: Removed LNI 192.168.201.12@tcp [ 735.098243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.150994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.165819] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 735.576258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[13-18/2]@tcp [ 735.594612] LNet: 31644:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 735.599431] LNet: 31644:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 735.621891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[13-18/2]@tcp [ 735.624965] LNetError: 31531:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.15@tcp has gone from up to down [ 735.661803] LNetError: 31531:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 741.897250] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:21:31 (1780449691) [ 742.239863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.265350] LNetError: 32236:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.283946] LNet: Removed LNI 192.168.201.12@tcp [ 742.348733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.471484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 742.592377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.683141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.823810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 742.890041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.918905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.958273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 743.008984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.041296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.094640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 743.180724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.217467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.297519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 743.396818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.438429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.530442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 743.648943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.684730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.754424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 743.860682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.937558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.012276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 744.088892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.150169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.216736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 744.328307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.373420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.423327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 744.484544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.522602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.596882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 744.650803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.697559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.761356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 750.992725] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:21:40 (1780449700) [ 752.836800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg112-server [ 754.632869] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:21:43 (1780449703) [ 756.537692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg112-server [ 758.039163] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:21:47 (1780449707) [ 759.113538] Key type .llcrypt unregistered [ 759.115755] Key type ._llcrypt unregistered [ 764.872528] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 764.999185] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 765.012236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 765.023191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 767.885431] Key type ._llcrypt registered [ 767.893030] Key type .llcrypt registered [ 768.056701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 768.071477] alg: No test for adler32 (adler32-zlib) [ 769.051595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.127832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 769.179913] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 769.914923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 769.938719] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 772.772591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 772.807696] LNet: Removed LNI 10.1.2.3@tcp [ 773.091286] LNet: 34360:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.099697] LNetError: 34360:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.121902] LNet: Removed LNI 192.168.201.12@tcp [ 773.900431] Key type .llcrypt unregistered [ 773.901957] Key type ._llcrypt unregistered [ 791.086060] Key type ._llcrypt registered [ 791.087820] Key type .llcrypt registered [ 792.506723] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:22:21 (1780449741) [ 793.037298] Key type .llcrypt unregistered [ 793.039612] Key type ._llcrypt unregistered [ 801.319827] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 801.423727] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 801.431881] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 801.438583] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 804.030988] Key type ._llcrypt registered [ 804.035594] Key type .llcrypt registered [ 804.175922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 804.183621] alg: No test for adler32 (adler32-zlib) [ 805.019434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.118322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 805.195579] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 806.120487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 806.176538] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 809.255927] LNet: 35856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.260923] LNetError: 35856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.271297] LNet: Removed LNI 192.168.201.12@tcp [ 809.288169] LNet: Removed LNI 10.1.2.3@tcp [ 809.996994] Key type .llcrypt unregistered [ 810.001339] Key type ._llcrypt unregistered [ 821.726346] Key type ._llcrypt registered [ 821.729241] Key type .llcrypt registered [ 822.800980] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:22:51 (1780449771) [ 823.371492] Key type .llcrypt unregistered [ 823.373842] Key type ._llcrypt unregistered [ 828.391430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 828.645218] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 828.652959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 828.662695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 831.244624] Key type ._llcrypt registered [ 831.249954] Key type .llcrypt registered [ 831.404526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 831.413212] alg: No test for adler32 (adler32-zlib) [ 832.296817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.340493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 832.359075] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 833.983756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 834.000814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.780534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 835.803644] LNet: Removed LNI 10.1.2.3@tcp [ 835.821815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 835.846961] LNet: Removed LNI 10.1.2.31@tcp [ 835.864712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 835.899288] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 836.740831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 836.763275] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 837.772569] LNet: 37776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.786549] LNetError: 37776:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 837.795911] LNet: Removed LNI 10.1.2.3@tcp [ 837.806051] LNet: Removed LNI 10.1.2.31@tcp [ 838.671748] Key type .llcrypt unregistered [ 838.674759] Key type ._llcrypt unregistered [ 855.888904] Key type ._llcrypt registered [ 855.902948] Key type .llcrypt registered [ 857.712472] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:23:26 (1780449806) [ 858.522352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 858.545377] alg: No test for adler32 (adler32-zlib) [ 859.423942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.461921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 859.486039] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 860.685365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.716707] LNet: 38729:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.724854] LNetError: 38729:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.750961] LNet: Removed LNI 192.168.201.12@tcp [ 860.830377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.929590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.12@tcp [ 860.991368] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 862.223977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.12@tcp [ 862.269676] LNet: Removed LNI 192.168.201.12@tcp [ 862.353962] LNet: 38864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.086438] Key type .llcrypt unregistered [ 863.089271] Key type ._llcrypt unregistered [ 877.363376] Key type ._llcrypt registered [ 877.364948] Key type .llcrypt registered [ 878.843053] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:23:47 (1780449827) [ 881.387535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg112-server [ 884.385063] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:23:52 (1780449832) [ 885.093485] Key type .llcrypt unregistered [ 885.100279] Key type ._llcrypt unregistered [ 895.153896] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 895.372036] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 895.387059] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 895.391426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 898.083579] Key type ._llcrypt registered [ 898.085934] Key type .llcrypt registered [ 898.247679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 898.263798] alg: No test for adler32 (adler32-zlib) [ 899.174962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.242829] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 900.275717] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 902.453960] LNet: 40701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.463912] LNetError: 40701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.473876] LNet: Removed LNI 192.168.201.12@tcp1 [ 902.491675] LNet: Removed LNI 10.1.2.3@tcp2 [ 903.094362] Key type .llcrypt unregistered [ 903.095952] Key type ._llcrypt unregistered [ 916.477966] Key type ._llcrypt registered [ 916.483898] Key type .llcrypt registered [ 917.709394] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:24:26 (1780449866) [ 918.201479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.225392] alg: No test for adler32 (adler32-zlib) [ 919.098351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.190001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 919.233212] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 920.442904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.461299] LNet: 41672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.465545] LNetError: 41672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.481962] LNet: Removed LNI 192.168.201.12@tcp [ 920.501346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.559899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 920.602040] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 928.933861] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:24:37 (1780449877) [ 929.279987] LNet: 42263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.288796] LNetError: 42263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.307124] LNet: Removed LNI 192.168.201.12@tcp [ 930.243666] Key type .llcrypt unregistered [ 930.247813] Key type ._llcrypt unregistered [ 937.418432] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 937.734083] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 937.744818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 937.760145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 940.381928] Key type ._llcrypt registered [ 940.384437] Key type .llcrypt registered [ 940.551933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.567907] alg: No test for adler32 (adler32-zlib) [ 941.637483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.717160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 941.745483] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 942.544674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 942.576385] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 944.301176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 944.323030] LNet: 43003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.328410] LNetError: 43003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.333562] LNet: Removed LNI 192.168.201.12@tcp [ 944.346523] LNet: Removed LNI 10.1.2.3@tcp [ 944.374867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.467958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 944.496497] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 944.508760] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.117840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 945.146671] LNet: 43030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.156864] LNetError: 43030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.169591] LNetError: 43030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 945.190074] LNet: Removed LNI 192.168.201.12@tcp [ 945.202392] LNet: Removed LNI 10.1.2.3@tcp [ 945.254194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 945.332153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 945.369820] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 945.372996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 945.792990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 945.813796] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 946.915526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.940433] LNet: 43192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.946282] LNetError: 43192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.952956] LNetError: 43192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 946.959620] LNet: Removed LNI 192.168.201.12@tcp [ 946.965481] LNet: Removed LNI 10.1.2.3@tcp [ 946.974820] LNet: Removed LNI 192.168.201.12@tcp2 [ 946.996720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.053833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 947.080911] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 947.084616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.092060] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 954.952694] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:25:04 (1780449904) [ 955.547765] LNet: 43789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.552574] LNetError: 43789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.555832] LNetError: 43789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 955.566458] LNet: Removed LNI 192.168.201.12@tcp [ 955.570524] LNet: Removed LNI 10.1.2.3@tcp [ 955.592202] LNet: Removed LNI 192.168.201.12@tcp2 [ 956.269820] Key type .llcrypt unregistered [ 956.272207] Key type ._llcrypt unregistered [ 963.128465] Key type ._llcrypt registered [ 963.133306] Key type .llcrypt registered [ 963.519347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 963.539408] alg: No test for adler32 (adler32-zlib) [ 964.497816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.581613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.623235] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 965.704743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.725734] LNet: 44184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.733596] LNetError: 44184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.754639] LNet: Removed LNI 192.168.201.12@tcp [ 965.790320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.866516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 965.888300] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 966.385535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 966.403419] LNet: 44214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.407975] LNetError: 44214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 966.418588] LNet: Removed LNI 192.168.201.12@tcp [ 966.432596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 966.504505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 966.601525] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 967.222807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 967.259369] LNet: Removed LNI 192.168.201.12@tcp [ 967.291428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 967.373718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 967.426118] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 967.971347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 968.092590] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 968.108992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 968.121925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 970.648193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 970.673642] LNet: 44318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.689104] LNet: 44318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 970.698154] LNetError: 44318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.707229] LNetError: 44318:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 970.715819] LNet: Removed LNI 192.168.201.12@tcp [ 970.751377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 970.833614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 970.861293] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 978.469154] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:25:27 (1780449927) [ 978.824367] LNet: 44915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.833637] LNetError: 44915:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.864943] LNet: Removed LNI 192.168.201.12@tcp [ 979.661847] Key type .llcrypt unregistered [ 979.663487] Key type ._llcrypt unregistered [ 988.195984] Key type ._llcrypt registered [ 988.200382] Key type .llcrypt registered [ 988.516380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.529589] alg: No test for adler32 (adler32-zlib) [ 989.464782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.540731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 989.565679] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 990.276763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.302273] LNet: 45314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.308937] LNetError: 45314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.351684] LNet: Removed LNI 192.168.201.12@tcp [ 990.380291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.475184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 990.509907] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 998.496499] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:25:47 (1780449947) [ 998.658076] LNet: 45898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 998.679497] LNetError: 45898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 998.711165] LNet: Removed LNI 192.168.201.12@tcp [ 999.738918] Key type .llcrypt unregistered [ 999.742798] Key type ._llcrypt unregistered [ 1006.979168] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1007.090324] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1007.101220] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.110478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1009.673070] Key type ._llcrypt registered [ 1009.674735] Key type .llcrypt registered [ 1009.883929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.911994] alg: No test for adler32 (adler32-zlib) [ 1010.839769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.911504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.951026] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1011.782974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.800222] LNet: 46357:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.805403] LNetError: 46357:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.819628] LNet: Removed LNI 192.168.201.12@tcp [ 1011.833415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.887840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1011.906376] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1017.988162] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:26:07 (1780449967) [ 1018.213245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.246594] LNet: 46955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.252874] LNetError: 46955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.273184] LNet: Removed LNI 192.168.201.12@tcp [ 1018.291308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.400473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1024.750652] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:26:13 (1780449973) [ 1025.061815] LNet: 47537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1025.788668] Key type .llcrypt unregistered [ 1025.794237] Key type ._llcrypt unregistered [ 1033.269655] Key type ._llcrypt registered [ 1033.273745] Key type .llcrypt registered [ 1033.433060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.457430] alg: No test for adler32 (adler32-zlib) [ 1034.463462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.525876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.542631] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1035.269763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.288333] LNet: 47934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.293781] LNetError: 47934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.305352] LNet: Removed LNI 192.168.201.12@tcp [ 1035.320309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.376803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1035.403069] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1042.610282] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:26:31 (1780449991) [ 1042.873756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1042.904098] LNet: 48528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.910573] LNetError: 48528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.923658] LNet: Removed LNI 192.168.201.12@tcp [ 1042.945294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1042.997948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1050.104158] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:26:38 (1780449998) [ 1052.475871] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1054.498701] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:26:43 (1780450003) [ 1055.947783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1057.726138] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:26:47 (1780450007) [ 1059.206569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1060.971819] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:26:50 (1780450010) [ 1062.660229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1064.339378] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:26:53 (1780450013) [ 1065.666691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1067.603114] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:26:56 (1780450016) [ 1067.894314] LNet: 50581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1068.740584] Key type .llcrypt unregistered [ 1068.742487] Key type ._llcrypt unregistered [ 1076.088105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1076.451522] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1076.454149] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1076.466268] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1079.001382] Key type ._llcrypt registered [ 1079.005319] Key type .llcrypt registered [ 1079.295544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.329385] alg: No test for adler32 (adler32-zlib) [ 1080.289443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.381532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1080.415683] LNet: Added LNI 192.168.201.12@tcp [8/512/0/180] [ 1080.428494] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1080.434210] LNet: 50902:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1080.553844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1080.574894] LNet: 50921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.580903] LNetError: 50921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.587438] LNet: Removed LNI 192.168.201.12@tcp [ 1080.596513] LNetError: 50921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1080.621113] LNet: Removed LNI 10.1.2.3@tcp [ 1080.644294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.710576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1080.744655] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1080.829512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1080.853484] LNet: 50946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.863127] LNetError: 50946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.891641] LNet: Removed LNI 192.168.201.12@tcp [ 1080.907148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.987464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1081.018087] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1081.027436] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1081.030357] LNet: 50954:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1081.141592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.161387] LNet: 50972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.173372] LNetError: 50972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.181676] LNet: Removed LNI 192.168.201.12@tcp [ 1081.184647] LNetError: 50972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1081.198283] LNet: Removed LNI 10.1.2.3@tcp [ 1081.222481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.307988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1081.340193] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1081.348105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1081.353444] LNet: 50980:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1081.640429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1081.656778] LNet: 51006:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.678971] LNetError: 51006:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.690587] LNet: Removed LNI 192.168.201.12@tcp [ 1081.699748] LNetError: 51006:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1081.725145] LNet: Removed LNI 10.1.2.3@tcp [ 1081.748543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.832398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1081.866580] LNet: Added LNI 192.168.201.12@tcp [8/256/0/182] [ 1081.870864] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1082.020444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1082.068103] LNet: 51034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.077767] LNetError: 51034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1082.085273] LNet: Removed LNI 192.168.201.12@tcp [ 1082.095394] LNetError: 51034:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1082.146685] LNet: Removed LNI 10.1.2.3@tcp [ 1082.177087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1082.237527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1082.264315] LNet: Added LNI 192.168.201.12@tcp [8/256/0/182] [ 1082.276359] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1082.278515] LNet: 51042:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1082.389601] LNet: Removed LNI 192.168.201.12@tcp [ 1082.410471] LNet: Removed LNI 10.1.2.3@tcp [ 1082.959107] Key type .llcrypt unregistered [ 1082.960849] Key type ._llcrypt unregistered [ 1096.929681] Key type ._llcrypt registered [ 1096.934422] Key type .llcrypt registered [ 1098.529649] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:27:27 (1780450047) [ 1099.165860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1099.177129] alg: No test for adler32 (adler32-zlib) [ 1100.220661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.271897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1100.303409] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1100.923393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.12 [ 1100.957146] LNet: 52017:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1100.965923] LNet: Removed LNI 192.168.201.12@tcp [ 1108.103562] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:27:37 (1780450057) [ 1108.328162] LNet: 52588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.347336] LNetError: 52588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.366362] LNet: Removed LNI 192.168.201.12@tcp [ 1109.053547] Key type .llcrypt unregistered [ 1109.059603] Key type ._llcrypt unregistered [ 1116.610693] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1116.912824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1119.574184] Key type ._llcrypt registered [ 1119.576546] Key type .llcrypt registered [ 1119.748341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1119.759224] alg: No test for adler32 (adler32-zlib) [ 1121.019679] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.328929] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:27:56 (1780450076) [ 1127.719232] LNet: 53503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.725675] LNetError: 53503:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1127.751321] LNet: Removed LNI 10.1.2.3@tcp [ 1128.449474] Key type .llcrypt unregistered [ 1128.456468] Key type ._llcrypt unregistered [ 1135.447623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1135.669727] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1135.677261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1135.686730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1138.400615] Key type ._llcrypt registered [ 1138.403595] Key type .llcrypt registered [ 1138.601069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1138.633372] alg: No test for adler32 (adler32-zlib) [ 1139.578391] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1146.229155] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:28:15 (1780450095) [ 1147.716669] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1149.800949] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:28:18 (1780450098) [ 1149.990749] LNet: 54686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.001813] LNetError: 54686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1150.028182] LNet: Removed LNI 10.1.2.3@tcp [ 1150.599131] Key type .llcrypt unregistered [ 1150.600770] Key type ._llcrypt unregistered [ 1157.533561] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1157.755226] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1157.757182] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1157.759337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1160.495337] Key type ._llcrypt registered [ 1160.498120] Key type .llcrypt registered [ 1160.667984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.691832] alg: No test for adler32 (adler32-zlib) [ 1161.675694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.746354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1169.809494] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:28:39 (1780450119) [ 1170.511183] LNet: 55918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.519464] LNetError: 55918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1170.547504] LNet: Removed LNI 10.1.2.3@tcp [ 1171.259667] Key type .llcrypt unregistered [ 1171.266117] Key type ._llcrypt unregistered [ 1178.204538] Key type ._llcrypt registered [ 1178.208621] Key type .llcrypt registered [ 1178.452271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1178.489971] alg: No test for adler32 (adler32-zlib) [ 1179.445689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1179.530438] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1189.749061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1190.811475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_interrupt -r 1 [ 1190.880323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_interrupt -r 1 [ 1190.934943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1190.958385] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1191.348322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.403290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1191.452742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.521098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.561416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.616344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.706496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.791225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.845708] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1191.848124] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1191.859504] LNet: 56357:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.886527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.944644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.978207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.012721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.045336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.080871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.109438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.086127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_dropped -r 1 [ 1193.210649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_dropped -r 1 [ 1193.313140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1193.338509] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1193.800867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.850398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1193.902491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.944635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1194.042096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1194.114628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1194.195836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1194.265991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1194.297122] LNet: 56467:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1194.336347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1194.399920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1194.459682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1194.513368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1194.607515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1194.686578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1194.761570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.606796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_aborted -r 1 [ 1195.653790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_aborted -r 1 [ 1195.696844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1195.720602] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1196.027532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1196.064844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1196.136849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1196.187690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1196.285687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1196.348860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1196.456690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1196.514408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1196.555657] LNet: 56576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1196.621258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1196.678692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1196.730531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1196.766478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1196.812186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1196.848560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1196.887662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1197.377209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_no_route -r 1 [ 1197.404968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_no_route -r 1 [ 1197.432111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1197.670690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1197.697891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1197.722658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1197.766887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1197.797262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1197.828114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1197.868084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1197.904437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1197.942578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1197.984304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1198.020831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1198.055673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1198.102329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1198.145983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1198.203609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1198.780684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_timeout -r 1 [ 1198.816963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_timeout -r 1 [ 1198.850983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1199.233404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.282373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1199.327672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.357167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.411078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.464669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1199.521494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1199.589311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1199.622432] LNet: 56795:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1199.627077] LNet: 56795:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1199.651515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1199.671888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1199.702070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1199.729993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.756954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1199.779039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1199.849359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.626494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_error -r 1 [ 1200.671254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_error -r 1 [ 1200.707970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1200.727470] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1200.735241] LNet: Skipped 3 previous similar messages [ 1201.003987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1201.050932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1201.094799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1201.138437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1201.168462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1201.234113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1201.280118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1201.320163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1201.369994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1201.433794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1201.489227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1201.525034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.599644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1201.673275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1201.747930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1202.106213] LNet: 56945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.113290] LNetError: 56945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.179090] LNet: Removed LNI 192.168.201.12@tcp [ 1203.858480] Key type .llcrypt unregistered [ 1203.860752] Key type ._llcrypt unregistered [ 1225.687403] Key type ._llcrypt registered [ 1225.692784] Key type .llcrypt registered [ 1227.667642] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:29:35 (1780450175) [ 1228.853502] Key type .llcrypt unregistered [ 1228.858248] Key type ._llcrypt unregistered [ 1237.031281] Key type ._llcrypt registered [ 1237.034393] Key type .llcrypt registered [ 1237.177721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1237.191851] alg: No test for adler32 (adler32-zlib) [ 1238.100962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1238.129592] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1246.595782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1258.706764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.778453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.813613] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1260.932378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_interrupt -r 1 [ 1260.984549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_interrupt -r 1 [ 1261.045528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_interrupt -r 1 [ 1261.085895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_interrupt -r 1 [ 1261.134545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_interrupt -r 1 [ 1261.180501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_interrupt -r 1 [ 1261.230485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_interrupt -r 1 [ 1261.289154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_interrupt -r 1 [ 1261.325845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1261.338715] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1261.347972] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1261.633431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.690493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1261.739630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.788793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.833612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.912413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.960285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1262.003503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1262.028341] LNet: 58416:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1262.046773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1262.069917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1262.099517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1262.134432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.169232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.208323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.245462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1262.821991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_dropped -r 1 [ 1262.863424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_dropped -r 1 [ 1262.893129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_dropped -r 1 [ 1262.925673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_dropped -r 1 [ 1262.958503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_dropped -r 1 [ 1262.988539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_dropped -r 1 [ 1263.021551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_dropped -r 1 [ 1263.067617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_dropped -r 1 [ 1263.099935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1263.130912] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1263.141874] LNet: Skipped 2 previous similar messages [ 1263.348322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.376287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1263.405061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.442871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.482757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.551351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.577765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.598901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.609412] LNet: 58537:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.627494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.666515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.690911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.726260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.776572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.806278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.839753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1264.350223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_aborted -r 1 [ 1264.391914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_aborted -r 1 [ 1264.430836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_aborted -r 1 [ 1264.481972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_aborted -r 1 [ 1264.523946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_aborted -r 1 [ 1264.560966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_aborted -r 1 [ 1264.611261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_aborted -r 1 [ 1264.646395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_aborted -r 1 [ 1264.690906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1264.711731] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1264.718527] LNet: Skipped 2 previous similar messages [ 1264.951864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1264.988175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1265.018393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1265.048962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1265.085923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1265.107540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1265.130191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1265.152858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1265.170552] LNet: 58659:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1265.188863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1265.217164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1265.263607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1265.308275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1265.347354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1265.385345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1265.422694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1265.936150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_no_route -r 1 [ 1265.976567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_no_route -r 1 [ 1266.011361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_no_route -r 1 [ 1266.054847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_no_route -r 1 [ 1266.085444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_no_route -r 1 [ 1266.123930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_no_route -r 1 [ 1266.164825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_no_route -r 1 [ 1266.204705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_no_route -r 1 [ 1266.243454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1266.448373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1266.472928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1266.494700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1266.524700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1266.567436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1266.601121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1266.643338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1266.698511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1266.728622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1266.759442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1266.790913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1266.818107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1266.851699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1266.882890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1266.921761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1267.466599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_timeout -r 1 [ 1267.516984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_timeout -r 1 [ 1267.596255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_timeout -r 1 [ 1267.644127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_timeout -r 1 [ 1267.690228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_timeout -r 1 [ 1267.718509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_timeout -r 1 [ 1267.759386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_timeout -r 1 [ 1267.823880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_timeout -r 1 [ 1267.880027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1267.898699] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1267.904152] LNet: Skipped 5 previous similar messages [ 1268.148591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.186221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1268.249328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.288963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.324426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1268.386855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1268.446073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1268.485172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1268.503297] LNet: 58901:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1268.511222] LNet: 58901:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1268.529054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1268.568831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1268.612678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1268.649599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1268.696262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.739814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1268.780940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1269.406397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_error -r 1 [ 1269.448246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_error -r 1 [ 1269.493884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_error -r 1 [ 1269.538376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_error -r 1 [ 1269.602514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_error -r 1 [ 1269.658198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_error -r 1 [ 1269.700352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_error -r 1 [ 1269.739951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_error -r 1 [ 1269.768334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1270.043088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1270.087978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1270.128668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1270.154615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1270.178362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1270.198373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1270.238228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1270.281802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1270.304219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1270.327585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1270.375034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1270.428677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1270.482076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1270.541575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1270.612678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.393315] LNet: 59099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.405739] LNetError: 59099:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1275.413276] LNet: Removed LNI 192.168.201.12@tcp [ 1275.426313] LNet: Removed LNI 192.168.201.12@tcp1 [ 1275.992722] Key type .llcrypt unregistered [ 1275.997203] Key type ._llcrypt unregistered [ 1295.722506] Key type ._llcrypt registered [ 1295.724721] Key type .llcrypt registered [ 1296.735286] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:30:46 (1780450246) [ 1297.682332] Key type .llcrypt unregistered [ 1297.688106] Key type ._llcrypt unregistered [ 1304.202518] Key type ._llcrypt registered [ 1304.210484] Key type .llcrypt registered [ 1304.544934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1304.573312] alg: No test for adler32 (adler32-zlib) [ 1305.522365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1305.563221] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1313.875881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1314.451123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_dropped -r 1 [ 1314.481485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_dropped -r 1 [ 1314.532907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1314.555534] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1314.836326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1314.885171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1314.951450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1314.995158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1315.031756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1315.066923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1315.112429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1315.153854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1315.195533] LNet: 60350:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1315.222619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1315.265884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1315.297652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.334055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1315.416861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1315.506394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1315.541446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1316.147139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_error -r 1 [ 1316.214930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_error -r 1 [ 1316.276353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1316.316141] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1316.590991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1316.630275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1316.654428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1316.679426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1316.705896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1316.732309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1316.763452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1316.790748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1316.807177] LNet: 60460:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.820303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1316.845714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1316.877880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.915819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1316.940333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1316.966801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1316.995330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1317.501791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_timeout -r 1 [ 1317.550711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_timeout -r 1 [ 1317.626502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1317.671866] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1318.039818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1318.097742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1318.151322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1318.181141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1318.213544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1318.252922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1318.310116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1318.352905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1318.375766] LNet: 60569:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1318.390872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1318.429235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1318.483937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1318.510342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1318.551063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1318.581938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1318.610880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1318.890588] LNet: 60610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1318.896938] LNetError: 60610:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1318.916213] LNet: Removed LNI 192.168.201.12@tcp [ 1319.342437] Key type .llcrypt unregistered [ 1319.344556] Key type ._llcrypt unregistered [ 1335.016462] Key type ._llcrypt registered [ 1335.017911] Key type .llcrypt registered [ 1336.370883] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:31:25 (1780450285) [ 1337.192490] Key type .llcrypt unregistered [ 1337.194431] Key type ._llcrypt unregistered [ 1343.949703] Key type ._llcrypt registered [ 1343.955319] Key type .llcrypt registered [ 1344.171626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.190839] alg: No test for adler32 (adler32-zlib) [ 1345.168208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1345.210376] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1353.654479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1363.058917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.152603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1363.187877] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1364.715809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_dropped -r 1 [ 1364.754131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e remote_dropped -r 1 [ 1364.790773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_dropped -r 1 [ 1364.828178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e remote_dropped -r 1 [ 1364.868647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e remote_dropped -r 1 [ 1364.901657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e remote_dropped -r 1 [ 1364.933114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e remote_dropped -r 1 [ 1364.993544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e remote_dropped -r 1 [ 1365.032593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1365.050958] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1365.058744] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.112@tcp [ 1365.317847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1365.353511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1365.389683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1365.414316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1365.441310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1365.468777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1365.496448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1365.529337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1365.545181] LNet: 62077:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1365.562746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1365.589336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1365.615719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1365.670291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.703687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1365.728716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1365.755334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1365.780947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1366.365429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_error -r 1 [ 1366.412333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e remote_error -r 1 [ 1366.459472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_error -r 1 [ 1366.493962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e remote_error -r 1 [ 1366.536541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e remote_error -r 1 [ 1366.579266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e remote_error -r 1 [ 1366.639217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e remote_error -r 1 [ 1366.702025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e remote_error -r 1 [ 1366.769950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1366.793781] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1366.798907] LNet: Skipped 2 previous similar messages [ 1367.128467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1367.169050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1367.209400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1367.253520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1367.323455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1367.361597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1367.422830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1367.470767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1367.493455] LNet: 62201:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1367.525833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1367.573949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1367.652630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1367.681168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1367.706741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1367.738934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.774585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1367.812299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1368.357489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_timeout -r 1 [ 1368.398669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e remote_timeout -r 1 [ 1368.430673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_timeout -r 1 [ 1368.460808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e remote_timeout -r 1 [ 1368.488794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e remote_timeout -r 1 [ 1368.529580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e remote_timeout -r 1 [ 1368.572506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e remote_timeout -r 1 [ 1368.611689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e remote_timeout -r 1 [ 1368.649555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.112@tcp [ 1368.675944] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1368.692242] LNet: Skipped 1 previous similar message [ 1368.896641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1368.939960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1368.966990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1368.993669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1369.020470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1369.050970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1369.083669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1369.109464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1369.125390] LNet: 62324:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.136742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1369.166304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1369.202435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1369.228693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1369.261720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1369.288289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1369.321975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1369.353629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1374.655055] LNet: 62402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.659384] LNetError: 62402:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.675515] LNet: Removed LNI 192.168.201.12@tcp [ 1374.707798] LNet: Removed LNI 192.168.201.12@tcp1 [ 1375.176681] Key type .llcrypt unregistered [ 1375.182235] Key type ._llcrypt unregistered [ 1393.834346] Key type ._llcrypt registered [ 1393.840368] Key type .llcrypt registered [ 1394.931708] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:32:24 (1780450344) [ 1396.246149] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1397.936697] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:32:27 (1780450347) [ 1398.792652] Key type .llcrypt unregistered [ 1398.794728] Key type ._llcrypt unregistered [ 1405.006398] Key type ._llcrypt registered [ 1405.009685] Key type .llcrypt registered [ 1405.229866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1405.251327] alg: No test for adler32 (adler32-zlib) [ 1406.155645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1406.179494] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1413.375211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1414.130423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e network_timeout -r 1 [ 1414.153799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e network_timeout -r 1 [ 1414.180622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1414.207765] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1414.369264] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1414.398646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1414.441163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1414.455764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1414.484636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1414.510351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1414.533746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1414.556476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1414.590670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1414.605508] LNet: 63954:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.633287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1414.672978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1414.695805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1414.740458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1414.782622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1414.810777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.031695] LNet: 63993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.035899] LNetError: 63993:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.050166] LNet: Removed LNI 192.168.201.12@tcp [ 1415.397572] Key type .llcrypt unregistered [ 1415.400476] Key type ._llcrypt unregistered [ 1433.991409] Key type ._llcrypt registered [ 1433.994172] Key type .llcrypt registered [ 1434.213350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.234417] alg: No test for adler32 (adler32-zlib) [ 1435.315911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1435.363442] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1442.840326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1452.886268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.953900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1453.000674] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1454.366506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e network_timeout -r 1 [ 1454.407488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e network_timeout -r 1 [ 1454.445537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e network_timeout -r 1 [ 1454.478835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e network_timeout -r 1 [ 1454.514982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e network_timeout -r 1 [ 1454.548498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e network_timeout -r 1 [ 1454.604095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e network_timeout -r 1 [ 1454.633887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e network_timeout -r 1 [ 1454.677195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1454.693607] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1454.815081] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1454.823480] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.112@tcp [ 1454.930421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1454.972763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1455.007594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1455.083126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1455.130294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1455.194512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1455.230533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1455.282230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1455.311238] LNet: 64874:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1455.343246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1455.423197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1455.469437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1455.505174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1455.538406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1455.586096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1460.736616] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1461.111656] LNet: 64948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.119617] LNetError: 64948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.124684] LNet: Removed LNI 192.168.201.12@tcp [ 1461.133614] LNet: Removed LNI 192.168.201.12@tcp1 [ 1461.537767] Key type .llcrypt unregistered [ 1461.541782] Key type ._llcrypt unregistered [ 1478.221802] Key type ._llcrypt registered [ 1478.228879] Key type .llcrypt registered [ 1479.312580] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:33:48 (1780450428) [ 1479.739854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1479.749756] alg: No test for adler32 (adler32-zlib) [ 1480.767299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.847741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1480.882327] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1481.392420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.408418] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1482.158647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 1482.215996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1482.247193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1482.296348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1482.352392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1482.390442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1482.440294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.12@tcp [ 1482.457367] LNetError: 66048:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.12@tcp to recovery [ 1482.847782] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1482.848461] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1484.896041] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1485.945972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1488.995990] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1489.444974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1489.489765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1489.545780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1489.569710] LNet: 66155:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.582291] LNetError: 66155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.655165] LNet: Removed LNI 192.168.201.12@tcp [ 1490.657154] LNetError: 66155:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.680096] LNet: Removed LNI 192.168.201.12@tcp1 [ 1490.704367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1490.762196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1490.795213] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1491.501562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1491.528526] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1492.287337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 1492.361096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1492.455620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1492.495847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1492.531205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1492.572506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1492.620390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1492.665124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.12@tcp [ 1492.686266] LNetError: 66428:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.12@tcp to recovery [ 1492.831514] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1495.224589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1498.979700] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1498.987862] LNet: Skipped 1 previous similar message [ 1502.365244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1502.421662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1502.471690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1510.024173] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:34:19 (1780450459) [ 1510.428585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.449749] LNet: 67126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.461719] LNetError: 67126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.467774] LNet: Removed LNI 192.168.201.12@tcp [ 1510.485392] LNet: Removed LNI 192.168.201.12@tcp1 [ 1510.514975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.609416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.658425] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1511.485741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1511.499649] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1512.475158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 1512.560919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1512.607387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1512.657751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1512.695864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1512.724527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1512.747720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.12@tcp --health 0 [ 1512.839625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1513.632425] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.12@tcp [ 1513.635411] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1513.656881] LNet: Skipped 1 previous similar message [ 1528.294442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1528.572093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1528.611334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1528.670547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.12@tcp --health 500 [ 1528.755550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1535.548122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1535.667649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.695983] LNet: 67618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1535.710267] LNetError: 67618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.719467] LNetError: 67618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1535.724551] LNet: Removed LNI 192.168.201.12@tcp [ 1535.738335] LNet: Removed LNI 192.168.201.12@tcp1 [ 1535.762204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.818385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.839772] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1536.538520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1536.575687] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1537.496956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 1537.563175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1537.644079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1537.667327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1537.707316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1537.747859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1537.782595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1537.820154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.12@tcp --health 0 [ 1538.847903] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1538.868067] LNet: Skipped 3 previous similar messages [ 1539.270091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.711599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1548.740608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1548.767779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1555.766867] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:35:04 (1780450504) [ 1556.275991] LNet: 68653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.281240] LNetError: 68653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.287386] LNetError: 68653:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1556.293158] LNet: Removed LNI 192.168.201.12@tcp [ 1556.303655] LNet: Removed LNI 192.168.201.12@tcp1 [ 1556.812446] Key type .llcrypt unregistered [ 1556.816379] Key type ._llcrypt unregistered [ 1563.854900] Key type ._llcrypt registered [ 1563.857148] Key type .llcrypt registered [ 1564.164855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.199445] alg: No test for adler32 (adler32-zlib) [ 1565.211558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1565.257624] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1575.560543] Lustre: DEBUG MARKER: Initial discovery [ 1575.716686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.112@tcp [ 1578.254324] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1578.664145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1578.734628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.112@tcp [ 1578.761476] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1578.788226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1578.834833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.112@tcp -r 1 -m GET -l 3 [ 1578.865356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.112@tcp -r 1 -m GET -e local_error [ 1578.906879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.112@tcp -r 1 -m PUT -l 6 [ 1579.625251] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1580.541906] Lustre: DEBUG MARKER: Force oleg112-client.virtnet to discover 192.168.201.112@tcp (in background) [ 1580.647570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.112@tcp [ 1581.899236] Lustre: DEBUG MARKER: Force oleg112-server to discover 192.168.201.12@tcp [ 1583.716273] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1584.243995] Lustre: DEBUG MARKER: Wait for 69410 [ 1585.796093] Lustre: DEBUG MARKER: Finished wait on 69410 [ 1586.084260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1586.132634] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1586.143752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1586.396624] LNet: 69867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.400886] LNetError: 69867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1586.417172] LNet: Removed LNI 192.168.201.12@tcp [ 1586.612550] Key type .llcrypt unregistered [ 1586.613716] Key type ._llcrypt unregistered [ 1604.580469] Key type ._llcrypt registered [ 1604.586593] Key type .llcrypt registered [ 1606.221334] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:35:54 (1780450554) [ 1606.922308] Key type .llcrypt unregistered [ 1606.927786] Key type ._llcrypt unregistered [ 1614.055307] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1614.106750] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1614.125090] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1614.133191] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1616.840770] Key type ._llcrypt registered [ 1616.847700] Key type .llcrypt registered [ 1617.128716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1617.193183] alg: No test for adler32 (adler32-zlib) [ 1618.132212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.173760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.191440] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1618.829509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1618.853935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1626.694801] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:36:16 (1780450576) [ 1627.064576] LNet: 71945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.077657] LNetError: 71945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.087532] LNet: Removed LNI 192.168.201.12@tcp [ 1627.105655] LNet: Removed LNI 10.1.2.3@tcp [ 1627.867433] Key type .llcrypt unregistered [ 1627.869994] Key type ._llcrypt unregistered [ 1634.884127] Key type ._llcrypt registered [ 1634.887814] Key type .llcrypt registered [ 1635.148712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1635.165955] alg: No test for adler32 (adler32-zlib) [ 1636.078859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.146494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1636.173518] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1637.091399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1637.107449] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1638.316198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.12@tcp [ 1639.392501] LNetError: Unexpected error -22 connecting to 192.168.201.12@tcp at host 192.168.201.12:988 [ 1645.274055] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:36:34 (1780450594) [ 1645.379347] LNet: 73087:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.387401] LNetError: 73087:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.393303] LNet: Removed LNI 192.168.201.12@tcp [ 1645.415034] LNet: Removed LNI 10.1.2.3@tcp [ 1646.231080] Key type .llcrypt unregistered [ 1646.238600] Key type ._llcrypt unregistered [ 1653.262415] Key type ._llcrypt registered [ 1653.279365] Key type .llcrypt registered [ 1653.530653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1653.553408] alg: No test for adler32 (adler32-zlib) [ 1654.489557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.565233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1654.598244] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1655.625520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1655.664978] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 1656.948920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.12@tcp1 --nid 192.168.201.12@tcp2 [ 1661.366967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp1 192.168.201.12@tcp2 [ 1661.385330] LNetError: 73930:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.12@tcp1 to 192.168.201.12@tcp2 - Route table may be misconfigured [ 1661.407672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp2 192.168.201.12@tcp1 [ 1667.857854] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:36:56 (1780450616) [ 1668.017090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.060728] LNet: 74498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.066769] LNetError: 74498:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.078913] LNet: Removed LNI 192.168.201.12@tcp1 [ 1669.102128] LNet: Removed LNI 192.168.201.12@tcp2 [ 1669.158768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.267871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.301854] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1670.038630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1670.051462] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1670.729427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 1670.761810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.12@tcp [ 1670.816214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp1 192.168.201.12@tcp1 [ 1670.858568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.12@tcp -d 192.168.201.12@tcp -e network_timeout [ 1670.905062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -e network_timeout [ 1670.947281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -e network_timeout [ 1670.986554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -e network_timeout [ 1671.036816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1671.071455] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1671.088471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1671.202867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1671.265195] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1671.344549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1677.850558] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:37:06 (1780450626) [ 1678.109341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.153607] LNet: 75365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.161045] LNetError: 75365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.166152] LNetError: 75365:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1678.172097] LNet: Removed LNI 192.168.201.12@tcp [ 1679.208656] LNet: Removed LNI 192.168.201.12@tcp1 [ 1679.245911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.344829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1679.719547] LNet: 75403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.109880] Key type .llcrypt unregistered [ 1680.111713] Key type ._llcrypt unregistered [ 1692.440384] Key type ._llcrypt registered [ 1692.444941] Key type .llcrypt registered [ 1693.885732] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:37:22 (1780450642) [ 1694.395310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1694.422081] alg: No test for adler32 (adler32-zlib) [ 1695.371176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.913826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1698.762286] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:37:27 (1780450647) [ 1698.963235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.991284] LNet: 76501:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.013886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.113690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1699.155467] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1699.766589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1699.787671] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1700.512278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1700.566417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp1 [ 1700.645836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp1 [ 1706.597890] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:37:35 (1780450655) [ 1708.136666] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg112-server [ 1709.808076] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:37:38 (1780450658) [ 1711.001410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg112-server [ 1712.517399] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:37:41 (1780450661) [ 1714.306815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg112-server [ 1715.666149] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:37:45 (1780450665) [ 1717.758915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg112-server [ 1719.441235] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:37:48 (1780450668) [ 1720.957211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg112-server [ 1722.572724] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:37:51 (1780450671) [ 1724.445305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg112-server [ 1726.193343] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:37:55 (1780450675) [ 1728.375615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg112-server [ 1730.066567] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:37:59 (1780450679) [ 1731.719379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg112-server [ 1733.777662] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:38:02 (1780450682) [ 1735.906426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg112-server [ 1738.120289] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:38:06 (1780450686) [ 1738.439631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.472175] LNet: 80379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.485899] LNetError: 80379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.551758] LNet: Removed LNI 192.168.201.12@tcp [ 1739.556291] LNetError: 80379:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.581750] LNet: Removed LNI 192.168.201.12@tcp1 [ 1739.662643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.727628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1739.764847] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1740.541713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1740.670837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1740.936141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.968901] LNet: 80527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.976718] LNetError: 80527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1741.000023] LNet: Removed LNI 192.168.201.12@tcp [ 1741.024932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.093315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1741.149101] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1741.987472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1742.184267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1743.270350] LNet: Removed LNI 192.168.201.12@tcp [ 1743.291574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.410850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1743.454752] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1744.131685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1744.244379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1744.432357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.446985] LNet: 80819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1744.451306] LNet: 80819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1744.455502] LNetError: 80819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1744.460037] LNetError: 80819:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1744.474963] LNet: Removed LNI 192.168.201.12@tcp [ 1744.500107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1744.566177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1744.604352] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1745.470930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1745.646458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.725072] LNet: Removed LNI 192.168.201.12@tcp [ 1746.762781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.866485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1746.899550] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1747.583918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1747.647672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1747.801049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.839748] LNet: Removed LNI 192.168.201.12@tcp [ 1748.855812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.969049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1749.015351] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1749.899884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1750.036935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.053766] LNet: 81257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.058526] LNet: 81257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1750.064737] LNetError: 81257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.071435] LNetError: 81257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1750.099041] LNet: Removed LNI 192.168.201.12@tcp [ 1750.116436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.152786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.172568] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1750.695558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1750.762102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1750.928753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.973944] LNet: Removed LNI 192.168.201.12@tcp [ 1751.991383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.057944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1752.077157] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1752.655425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1752.795592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.830744] LNet: Removed LNI 192.168.201.12@tcp [ 1753.885962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.973560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.010762] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1754.756883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1754.819465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1754.980191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1756.005751] LNet: Removed LNI 192.168.201.12@tcp [ 1756.058975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.125917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1756.165455] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1757.129171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1758.630491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.660101] LNet: 81851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1758.679410] LNet: 81851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1758.699846] LNetError: 81851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.710728] LNetError: 81851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1758.742440] LNet: Removed LNI 192.168.201.12@tcp [ 1758.776299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.874846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.904457] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1759.882127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1760.033804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1760.436323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1761.508917] LNet: Removed LNI 192.168.201.12@tcp [ 1761.581092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.687377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1761.751987] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1762.705523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 1763.077529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.135076] LNet: Removed LNI 192.168.201.12@tcp [ 1764.174687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.308718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.351959] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1765.135529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1765.196463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.225571] LNet: Removed LNI 192.168.201.12@tcp [ 1765.270864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.434186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1765.475705] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1766.436115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1773.875449] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:38:42 (1780450722) [ 1774.039320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1774.088379] LNet: Removed LNI 192.168.201.12@tcp [ 1774.116142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.249273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1774.294908] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/180] [ 1775.775375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.798429] LNet: 83119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.803800] LNet: 83119:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1775.809918] LNetError: 83119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1775.816055] LNetError: 83119:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1775.828795] LNet: Removed LNI 192.168.201.12@tcp231 [ 1775.849311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.950541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1775.973726] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/180] [ 1775.980243] LNet: 83127:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1776.857908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1776.936769] LNet: Removed LNI 192.168.201.12@tcp231 [ 1776.968894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1776.993522] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/180] [ 1778.312224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1778.367512] LNet: Removed LNI 192.168.201.12@tcp231 [ 1778.398237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1778.481663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1778.498805] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/0] [ 1778.502938] LNet: 83279:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1779.039805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1779.062205] LNet: Removed LNI 192.168.201.12@tcp231 [ 1779.086732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1779.126554] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/0] [ 1780.830759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1780.874572] LNet: Removed LNI 192.168.201.12@tcp231 [ 1780.905798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1780.987689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1781.021099] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/60] [ 1781.024413] LNet: 83430:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1781.782987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1781.845281] LNet: Removed LNI 192.168.201.12@tcp231 [ 1781.878847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1781.934719] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/60] [ 1783.053142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.087594] LNet: Removed LNI 192.168.201.12@tcp231 [ 1783.113557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.182695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1783.236831] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/180] [ 1783.249965] LNet: 83582:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1783.870319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1783.902410] LNet: Removed LNI 192.168.201.12@tcp231 [ 1783.932349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1783.990655] LNet: Added LNI 192.168.201.12@tcp231 [8/256/0/180] [ 1794.022875] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:39:02 (1780450742) [ 1794.148293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.194154] LNet: Removed LNI 192.168.201.12@tcp231 [ 1794.236232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.386147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1794.467489] LNet: Added LNI 192.168.201.12@tcp232 [8/256/0/180] [ 1795.361633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1795.418871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1795.449269] LNet: Removed LNI 192.168.201.12@tcp232 [ 1795.467781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1795.503913] LNet: Added LNI 192.168.201.12@tcp232 [8/256/0/180] [ 1796.359859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1796.440521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1796.508401] LNet: Removed LNI 192.168.201.12@tcp232 [ 1796.563686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1796.618253] LNet: Added LNI 192.168.201.12@tcp232 [8/256/0/180] [ 1797.646396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1797.701461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1797.806422] LNet: Removed LNI 192.168.201.12@tcp232 [ 1797.850433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1797.881770] LNet: Added LNI 192.168.201.12@tcp232 [8/256/0/180] [ 1798.558608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1798.591753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1798.621466] LNet: Removed LNI 192.168.201.12@tcp232 [ 1798.640514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1805.643392] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:39:14 (1780450754) [ 1806.819220] Key type .llcrypt unregistered [ 1806.822136] Key type ._llcrypt unregistered [ 1814.298165] Key type ._llcrypt registered [ 1814.300992] Key type .llcrypt registered [ 1814.482700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1814.507573] alg: No test for adler32 (adler32-zlib) [ 1815.484700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1815.537715] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1824.512153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1834.655572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1834.733381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1834.774485] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1836.313780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_interrupt -r 2 [ 1836.399698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_interrupt -r 2 [ 1836.474657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_interrupt -r 2 [ 1836.537062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_interrupt -r 2 [ 1836.596989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_interrupt -r 2 [ 1836.653250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_interrupt -r 2 [ 1836.693630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_interrupt -r 2 [ 1836.744867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_interrupt -r 2 [ 1836.774820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.112@tcp [ 1836.789316] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1836.802040] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1837.090720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1837.124913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1837.154747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1837.185440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1837.232570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1837.281868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1837.345950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1837.409456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1837.434971] LNet: 86082:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1837.456957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1837.486482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1837.531268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1837.568056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1837.591583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1837.618809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1838.160618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_dropped -r 2 [ 1838.183721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_dropped -r 2 [ 1838.212989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_dropped -r 2 [ 1838.240089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_dropped -r 2 [ 1838.269813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_dropped -r 2 [ 1838.301804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_dropped -r 2 [ 1838.337794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_dropped -r 2 [ 1838.374550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_dropped -r 2 [ 1838.424605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.112@tcp [ 1838.455978] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1838.822438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1838.878102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1838.926154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1838.967435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1839.013979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1839.050443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1839.089441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1839.138460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1839.163309] LNet: 86203:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1839.198944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1839.261173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1839.312370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1839.354304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1839.408510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1839.469765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1840.174622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_aborted -r 2 [ 1840.247391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_aborted -r 2 [ 1840.328675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_aborted -r 2 [ 1840.383932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_aborted -r 2 [ 1840.448104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_aborted -r 2 [ 1840.494374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_aborted -r 2 [ 1840.539641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_aborted -r 2 [ 1840.603750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_aborted -r 2 [ 1840.630411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.112@tcp [ 1840.647557] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1840.919408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1840.949382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1841.000058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1841.041477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1841.075983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1841.113236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1841.161186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1841.212242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1841.255978] LNet: 86323:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1841.281671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1841.326678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1841.372236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1841.419640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1841.461179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1841.508040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1842.068280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_no_route -r 2 [ 1842.129820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_no_route -r 2 [ 1842.192623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_no_route -r 2 [ 1842.232065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_no_route -r 2 [ 1842.273976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_no_route -r 2 [ 1842.333657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_no_route -r 2 [ 1842.394894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_no_route -r 2 [ 1842.440199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_no_route -r 2 [ 1842.486577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.112@tcp [ 1842.762814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1842.798503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1842.830283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1842.854135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1842.883561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1842.904761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1842.931240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1842.949505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1842.975855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1843.005725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1843.044642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1843.076614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1843.101575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1843.139872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1843.836707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e local_timeout -r 2 [ 1843.914338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e local_timeout -r 2 [ 1843.963552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e local_timeout -r 2 [ 1844.029462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e local_timeout -r 2 [ 1844.083509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e local_timeout -r 2 [ 1844.110816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e local_timeout -r 2 [ 1844.141636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e local_timeout -r 2 [ 1844.184898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e local_timeout -r 2 [ 1844.233818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.112@tcp [ 1844.254795] LNet: There was an unexpected network error while writing to 192.168.201.112: rc = -22 [ 1844.260801] LNet: Skipped 1 previous similar message [ 1844.603845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1844.645959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1844.676444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1844.720596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1844.778958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1844.836658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1844.883105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1844.934460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1844.965811] LNet: 86564:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1844.978872] LNet: 86564:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1845.029672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1845.095774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1845.132905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1845.168381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1845.201244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1845.235184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1845.660293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp -m GET -e remote_dropped -r 2 [ 1845.688325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp -d 192.168.201.112@tcp1 -m GET -e remote_dropped -r 2 [ 1845.732490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp -m GET -e remote_dropped -r 2 [ 1845.765683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp -d 192.168.201.12@tcp1 -m GET -e remote_dropped -r 2 [ 1845.800819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp -m GET -e remote_dropped -r 2 [ 1845.832605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.12@tcp1 -d 192.168.201.112@tcp1 -m GET -e remote_dropped -r 2 [ 1845.871303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp -m GET -e remote_dropped -r 2 [ 1845.908653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.12@tcp1 -d 192.168.201.12@tcp1 -m GET -e remote_dropped -r 2 [ 1845.947929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.12@tcp 192.168.201.112@tcp [ 1845.984585] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.112@tcp [ 1846.277128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1846.310729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1846.345687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1846.378642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1846.401379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1846.436621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1846.465381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1846.490501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1846.521803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1846.559090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1846.603437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1846.663266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1846.693819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1846.732110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1852.936914] LNet: 86758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.943956] LNetError: 86758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.952855] LNet: Removed LNI 192.168.201.12@tcp [ 1852.969124] LNet: Removed LNI 192.168.201.12@tcp1 [ 1853.338311] Key type .llcrypt unregistered [ 1853.340957] Key type ._llcrypt unregistered [ 1869.837331] Key type ._llcrypt registered [ 1869.841606] Key type .llcrypt registered [ 1871.059566] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:40:20 (1780450820) [ 1871.998246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1872.017234] alg: No test for adler32 (adler32-zlib) [ 1872.880532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.943645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.970615] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1873.720606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1873.747715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.12@tcp [ 1873.763267] LNetError: 87738:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.12@tcp to recovery [ 1873.867238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.12@tcp [ 1874.053115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1881.737904] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:40:30 (1780450830) [ 1881.984118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1882.044085] LNet: 88322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.050688] LNetError: 88322:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.096458] LNet: Removed LNI 192.168.201.12@tcp [ 1882.151679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.291526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1882.334381] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1883.628987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1883.657558] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 1885.176865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.12@tcp [ 1885.232189] LNetError: 88594:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.12@tcp to recovery [ 1885.282944] LNet: 1 local NIs in recovery (showing 1): 192.168.201.12@tcp [ 1885.295262] LNet: There was an unexpected network error while writing to 192.168.201.12: rc = -22 [ 1885.439700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.12@tcp [ 1896.387689] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:40:45 (1780450845) [ 1898.983370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg112-server [ 1901.488693] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:40:50 (1780450850) [ 1903.954756] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg112-server [ 1905.538293] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:40:54 (1780450854) [ 1907.930119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg112-server [ 1909.817854] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:40:58 (1780450858) [ 1909.960131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.998799] LNet: 90214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1910.004051] LNetError: 90214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.010517] LNet: Removed LNI 192.168.201.12@tcp [ 1910.034488] LNet: Removed LNI 192.168.201.12@tcp1 [ 1910.102764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.168912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.190902] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1911.187757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1911.257481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1911.374805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1911.510188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1911.624754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1911.683646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1918.703267] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:41:07 (1780450867) [ 1919.135740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.170525] LNet: 90943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.178701] LNetError: 90943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.193321] LNetError: 90943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1919.227597] LNet: Removed LNI 192.168.201.12@tcp [ 1919.262824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.330747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1919.364968] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1928.061410] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:41:16 (1780450876) [ 1928.242241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.276979] LNet: 91639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.293309] LNetError: 91639:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.332516] LNet: Removed LNI 192.168.201.12@tcp [ 1928.373277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1928.484883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1928.535667] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1937.055586] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:41:25 (1780450885) [ 1938.564868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1940.517885] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:41:29 (1780450889) [ 1941.270375] LNet: 92682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.280041] LNetError: 92682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.296982] LNet: Removed LNI 192.168.201.12@tcp [ 1941.954444] Key type .llcrypt unregistered [ 1941.956778] Key type ._llcrypt unregistered [ 1947.945826] Key type ._llcrypt registered [ 1947.948475] Key type .llcrypt registered [ 1948.107162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1948.134981] alg: No test for adler32 (adler32-zlib) [ 1949.087533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1949.130888] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 1958.436310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 1965.135157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.112@tcp [ 1965.626211] LNet: 93100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1965.631728] LNetError: 93100:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1965.643623] LNet: Removed LNI 192.168.201.12@tcp [ 1966.198821] Key type .llcrypt unregistered [ 1966.201546] Key type ._llcrypt unregistered [ 1978.958505] Key type ._llcrypt registered [ 1978.962996] Key type .llcrypt registered [ 1980.322841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1981.900631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1983.585723] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:42:12 (1780450932) [ 1985.577831] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1987.419203] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:42:16 (1780450936) [ 1989.180473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg112-server [ 1991.177751] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:42:20 (1780450940) [ 1993.501788] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg112-server [ 1995.870674] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:42:24 (1780450944) [ 1997.744986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.803957] alg: No test for adler32 (adler32-zlib) [ 1998.904223] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 1999.172130] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2000.872352] Key type lgssc registered [ 2002.394119] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2026.195972] Key type lgssc unregistered [ 2026.471819] LNet: 96277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.478821] LNetError: 96277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.502582] LNet: Removed LNI 192.168.201.12@tcp [ 2027.212796] Key type .llcrypt unregistered [ 2027.215664] Key type ._llcrypt unregistered [ 2043.002951] Key type ._llcrypt registered [ 2043.004722] Key type .llcrypt registered [ 2044.369708] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:43:13 (1780450993) [ 2045.111082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2045.128807] alg: No test for adler32 (adler32-zlib) [ 2046.017991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2046.108538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2046.190720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2046.304355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2046.382732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2053.258778] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:43:22 (1780451002) [ 2053.484881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.526547] LNet: 97692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.552444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.653744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.681543] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2054.342261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.357563] LNet: 97836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2054.362851] LNetError: 97836:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.393965] LNet: Removed LNI 192.168.201.12@tcp [ 2054.418612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.479497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2054.497940] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2054.886219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2054.903769] LNetError: 97864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2054.932058] LNet: Removed LNI 192.168.201.12@tcp [ 2054.954029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.995909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2055.022865] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2062.462517] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:43:31 (1780451011) [ 2062.574766] LNet: 98447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.583011] LNet: 98447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2062.591356] LNetError: 98447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2062.610152] LNet: Removed LNI 192.168.201.12@tcp [ 2063.452599] Key type .llcrypt unregistered [ 2063.457648] Key type ._llcrypt unregistered [ 2070.001571] Key type ._llcrypt registered [ 2070.003649] Key type .llcrypt registered [ 2070.145300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2070.161097] alg: No test for adler32 (adler32-zlib) [ 2071.112591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.189829] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2071.191923] LNet: *** cfs_fail_loc=e003, val=100*** [ 2071.713606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 2071.734748] LNet: *** cfs_fail_loc=e003, val=100*** [ 2071.739380] LNet: Skipped 2 previous similar messages [ 2071.745813] Ping reply received with mapping: [ 2071.748534] found metadata reply NI[192.168.201.12@tcp] [ 2071.751371] 0c c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2071.757726] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2071.762493] 5a 5a 5a 5a 5a 5a 5a 5a [ 2071.868719] LNet: 98852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.874882] LNetError: 98852:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.940391] LNet: Removed LNI 192.168.201.12@tcp [ 2073.407708] Key type .llcrypt unregistered [ 2073.409868] Key type ._llcrypt unregistered [ 2084.790956] Key type ._llcrypt registered [ 2084.793916] Key type .llcrypt registered [ 2086.018146] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:43:55 (1780451035) [ 2086.561395] Key type .llcrypt unregistered [ 2086.563345] Key type ._llcrypt unregistered [ 2093.959510] Key type ._llcrypt registered [ 2093.963843] Key type .llcrypt registered [ 2094.163567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2094.185209] alg: No test for adler32 (adler32-zlib) [ 2095.354568] LNetError: 99900:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2095.658367] Key type .llcrypt unregistered [ 2095.661404] Key type ._llcrypt unregistered [ 2100.683944] Key type ._llcrypt registered [ 2100.689248] Key type .llcrypt registered [ 2101.990874] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:44:11 (1780451051) [ 2102.597669] Key type .llcrypt unregistered [ 2102.600452] Key type ._llcrypt unregistered [ 2109.155639] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2109.163277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2111.657202] Key type ._llcrypt registered [ 2111.659105] Key type .llcrypt registered [ 2111.852338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2111.870535] alg: No test for adler32 (adler32-zlib) [ 2112.809246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.907257] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2121.004699] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:44:30 (1780451070) [ 2121.452719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.491021] LNet: 101682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.512957] LNetError: 101682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2121.549194] LNet: Removed LNI 10.1.2.3@tcp [ 2121.574133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.289569] Key type .llcrypt unregistered [ 2122.290918] Key type ._llcrypt unregistered [ 2128.394676] Key type ._llcrypt registered [ 2128.395967] Key type .llcrypt registered [ 2128.566612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2128.594791] alg: No test for adler32 (adler32-zlib) [ 2129.648494] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2129.771315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 2129.980733] LNet: 101971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.000539] LNetError: 101971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.020789] LNet: Removed LNI 192.168.201.12@tcp [ 2130.769138] Key type .llcrypt unregistered [ 2130.776468] Key type ._llcrypt unregistered [ 2145.649626] Key type ._llcrypt registered [ 2145.653689] Key type .llcrypt registered [ 2147.396125] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:44:55 (1780451095) [ 2148.458331] Key type .llcrypt unregistered [ 2148.460637] Key type ._llcrypt unregistered [ 2156.016079] Key type ._llcrypt registered [ 2156.017775] Key type .llcrypt registered [ 2156.281719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2156.329212] alg: No test for adler32 (adler32-zlib) [ 2164.417669] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:45:13 (1780451113) [ 2164.926748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.991374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.082951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2165.117764] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2165.980569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.017469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2166.050290] LNet: Removed LNI 192.168.201.12@tcp [ 2166.332964] LNet: 103717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2166.755626] Key type .llcrypt unregistered [ 2166.758379] Key type ._llcrypt unregistered [ 2182.863778] Key type ._llcrypt registered [ 2182.873605] Key type .llcrypt registered [ 2184.634141] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:45:33 (1780451133) [ 2185.105775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2185.119970] alg: No test for adler32 (adler32-zlib) [ 2186.124923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.189362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.212829] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2187.351701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 2187.416891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.12@tcp [ 2187.458305] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2187.468456] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2187.471878] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2187, 0, 0 [ 2187.481217] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2187.482914] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2187.493174] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2187.504984] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2187, 0, 0 [ 2187.518150] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2187.524783] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2187.531749] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2187.535603] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2187, 0, 2237 [ 2187.541761] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2187.547929] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2187.554465] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2187.557827] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2187, 0, 2237 [ 2187.569592] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2193.910181] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:45:43 (1780451143) [ 2194.316304] LNet: 105260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2194.319675] LNetError: 105260:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2194.334116] LNet: Removed LNI 192.168.201.12@tcp [ 2194.861483] Key type .llcrypt unregistered [ 2194.867456] Key type ._llcrypt unregistered [ 2201.070625] Key type ._llcrypt registered [ 2201.073592] Key type .llcrypt registered [ 2201.321859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2201.339770] alg: No test for adler32 (adler32-zlib) [ 2202.318237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2202.338299] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2212.267411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 2223.878931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2223.948776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2224.004840] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 2224.873347] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2225.021958] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2225.026446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2225.031545] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2227.381108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2227.406500] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2238.897833] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:46:28 (1780451188) [ 2239.551601] LNet: 106770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.566309] LNetError: 106770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.578586] LNet: Removed LNI 192.168.201.12@tcp [ 2240.607413] LNet: Removed LNI 192.168.201.12@tcp1 [ 2240.612039] LNetError: 106770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2240.628022] LNetError: 106770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2240.658564] LNet: Removed LNI 10.1.2.3@tcp99 [ 2241.280954] Key type .llcrypt unregistered [ 2241.283895] Key type ._llcrypt unregistered [ 2247.770454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2247.779360] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2250.467734] Key type ._llcrypt registered [ 2250.470545] Key type .llcrypt registered [ 2250.690510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2250.706315] alg: No test for adler32 (adler32-zlib) [ 2251.697631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.783479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.810720] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2252.726672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2252.764807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2255.078950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2255.160401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2255.198667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2255.215986] LNetError: 107515:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2255.278638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2255.358493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2255.835088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2255.881404] LNet: 107535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2255.885522] LNetError: 107535:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2255.888915] LNet: Removed LNI 192.168.201.12@tcp [ 2255.894642] LNet: Removed LNI 10.1.2.3@tcp [ 2256.454635] Key type .llcrypt unregistered [ 2256.455823] Key type ._llcrypt unregistered [ 2269.615689] Key type ._llcrypt registered [ 2269.617847] Key type .llcrypt registered [ 2270.935959] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:46:59 (1780451219) [ 2271.370798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2271.386626] alg: No test for adler32 (adler32-zlib) [ 2272.316992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.372376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2272.393336] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2273.150324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.12@tcp [ 2273.239887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg112-client@tcp [ 2279.100633] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:47:08 (1780451228) [ 2279.179445] LNet: 109059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.188586] LNetError: 109059:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.218309] LNet: Removed LNI 192.168.201.12@tcp [ 2279.777686] Key type .llcrypt unregistered [ 2279.782174] Key type ._llcrypt unregistered [ 2286.568681] Key type ._llcrypt registered [ 2286.570090] Key type .llcrypt registered [ 2286.792121] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.821922] alg: No test for adler32 (adler32-zlib) [ 2287.799306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2287.823072] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2287.857233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2294.989783] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:47:23 (1780451243) [ 2297.062794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2299.164762] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:47:27 (1780451247) [ 2300.846787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2302.666710] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:47:31 (1780451251) [ 2302.993252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.021056] LNet: 110488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.027976] LNetError: 110488:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.076077] LNet: Removed LNI 192.168.201.12@tcp [ 2303.138963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2303.237826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2303.281300] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2304.254774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2304.317764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2311.092636] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:47:40 (1780451260) [ 2311.188286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2311.231678] LNet: 111194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2311.242338] LNetError: 111194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2311.266396] LNet: Removed LNI 192.168.201.12@tcp [ 2311.314103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2311.402285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2311.458732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2312.002591] Key type .llcrypt unregistered [ 2312.008573] Key type ._llcrypt unregistered [ 2325.395649] Key type ._llcrypt registered [ 2325.397842] Key type .llcrypt registered [ 2326.948372] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:47:55 (1780451275) [ 2327.281304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2327.300167] alg: No test for adler32 (adler32-zlib) [ 2328.215829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2328.281189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2328.319169] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2328.924250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2328.975290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2335.525412] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:48:04 (1780451284) [ 2335.883917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2335.900313] LNet: 112753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2335.908455] LNetError: 112753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2336.935068] LNet: Removed LNI 192.168.201.12@tcp [ 2336.978484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2337.054379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2337.092675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2343.655432] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:48:12 (1780451292) [ 2343.777918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.797543] LNet: 113328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.836459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.938875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2344.343302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2344.398287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2344.459653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2344.919756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2344.951807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2345.008481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2345.031863] LNet: 113368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.059589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.118445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2345.625837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2345.658323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2345.708314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2346.061790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2346.094091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2352.885720] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:48:21 (1780451301) [ 2353.092812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2353.129237] LNet: 113968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2353.151931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.222096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2353.244540] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2353.995128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2354.041161] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2355.075852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2355.131768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2355.499784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.548559] LNetError: 114247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.563351] LNet: Removed LNI 192.168.201.12@tcp [ 2355.584723] LNet: Removed LNI 192.168.201.12@tcp2 [ 2355.615943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.714836] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2355.723665] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2355.732606] LNet: 114253:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2356.465591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2356.758727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.773588] LNetError: 114291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.778022] LNetError: 114291:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2356.781611] LNet: Removed LNI 192.168.201.12@tcp [ 2356.794568] LNet: Removed LNI 192.168.201.12@tcp2 [ 2356.820935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.945595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.985860] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2357.704329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2357.724770] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2358.628213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2358.653480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2358.721501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2358.908677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.934469] LNet: 114575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.942371] LNet: 114575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2358.948926] LNetError: 114575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.952856] LNetError: 114575:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2358.961490] LNet: Removed LNI 192.168.201.12@tcp [ 2359.981074] LNet: Removed LNI 192.168.201.12@tcp2 [ 2360.051132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.141724] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2360.152172] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2360.163864] LNet: 114581:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2361.077268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2361.476862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.518196] LNet: Removed LNI 192.168.201.12@tcp [ 2362.600230] LNet: Removed LNI 192.168.201.12@tcp2 [ 2362.687568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.833907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2362.897628] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2363.861256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2363.897648] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2365.415767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2365.480618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2365.745384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.776740] LNetError: 114898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2365.783361] LNetError: 114898:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2365.800923] LNet: Removed LNI 192.168.201.12@tcp [ 2365.812223] LNet: Removed LNI 192.168.201.12@tcp2 [ 2365.852321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.939927] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2365.945116] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2365.951152] LNet: 114904:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.530427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2366.854821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.903453] LNet: Removed LNI 192.168.201.12@tcp [ 2367.920626] LNet: Removed LNI 192.168.201.12@tcp2 [ 2367.999742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.065940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2368.088082] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2369.075934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2369.111811] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2370.557164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2370.628805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2370.756850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2371.131853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.168656] LNet: 115226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.180767] LNet: 115226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2372.255607] LNet: Removed LNI 192.168.201.12@tcp [ 2373.350365] LNet: Removed LNI 192.168.201.12@tcp2 [ 2373.456145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.608312] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2373.615465] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2373.627688] LNet: 115232:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.972433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2375.435693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2375.494702] LNetError: 115269:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2375.509494] LNetError: 115269:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2375.531489] LNet: Removed LNI 192.168.201.12@tcp [ 2376.560332] LNet: Removed LNI 192.168.201.12@tcp2 [ 2376.600760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.675703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2376.716601] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2377.732334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2377.765414] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2378.802560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2378.872425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2378.969860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2379.508402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2380.576879] LNet: Removed LNI 192.168.201.12@tcp [ 2380.599740] LNet: Removed LNI 192.168.201.12@tcp2 [ 2380.664743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.818162] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2380.839433] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2380.851809] LNet: 115572:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2381.402710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2381.798655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.879397] LNet: Removed LNI 192.168.201.12@tcp [ 2382.890949] LNet: Removed LNI 192.168.201.12@tcp2 [ 2382.915945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.970782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.992165] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2383.558925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2383.584686] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2384.530872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2384.563570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2384.595943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2384.647516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2385.015632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.080280] LNet: Removed LNI 192.168.201.12@tcp [ 2387.113056] LNet: Removed LNI 192.168.201.12@tcp2 [ 2387.175448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.249073] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2387.254720] LNet: Added LNI 192.168.201.12@tcp2 [8/256/0/180] [ 2387.865084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2396.033861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2397.694855] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:49:06 (1780451346) [ 2398.042110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2398.068087] LNet: 116691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2398.074459] LNet: 116691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2398.082276] LNetError: 116691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2398.085161] LNetError: 116691:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2398.088938] LNet: Removed LNI 192.168.201.12@tcp [ 2399.143265] LNet: Removed LNI 192.168.201.12@tcp2 [ 2399.178323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.561302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2399.610278] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2400.313449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2400.329393] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2401.088417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2401.104746] LNet: Removed LNI 192.168.201.12@tcp [ 2401.112891] LNet: Removed LNI 10.1.2.3@tcp [ 2401.140300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.203252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2401.232764] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2401.237602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2401.738930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2401.772322] LNet: Removed LNI 192.168.201.12@tcp [ 2401.785491] LNet: Removed LNI 10.1.2.3@tcp [ 2408.870669] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:49:17 (1780451357) [ 2409.016510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2409.090901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2409.227575] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2409.334131] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2409.339249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2409.346928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2411.687842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2411.716706] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2412.423614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2412.452793] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2414.724910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.12@tcp [ 2414.754470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2414.797846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.12@tcp -m PUT -e local_timeout -r 1 [ 2414.841925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2414.886252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2414.888696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2415.131814] LNetError: 117629:0:(lib-move.c:4710:lnet_parse()) 192.168.201.12@tcp, src 192.168.201.12@tcp: Dropping PUT (error -108 looking up sender) [ 2417.183171] LNet: Removed LNI 192.168.201.12@tcp [ 2417.203475] LNet: Removed LNI 10.1.2.3@tcp [ 2417.757987] Key type .llcrypt unregistered [ 2417.761665] Key type ._llcrypt unregistered [ 2434.226777] Key type ._llcrypt registered [ 2434.231452] Key type .llcrypt registered [ 2435.994401] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:49:44 (1780451384) [ 2436.467407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2436.495679] alg: No test for adler32 (adler32-zlib) [ 2437.489961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2438.147816] LNet: 118944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2438.933382] Key type .llcrypt unregistered [ 2438.935555] Key type ._llcrypt unregistered [ 2445.779962] Key type ._llcrypt registered [ 2445.788348] Key type .llcrypt registered [ 2445.970924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2446.007936] alg: No test for adler32 (adler32-zlib) [ 2446.928434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2446.963134] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2455.625443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 2455.842100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2456.668869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.112@tcp [ 2456.831800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2456.856958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2456.883398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2456.920647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2456.970507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2457.016157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2457.044879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2457.072986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2457.087318] LNet: 119395:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2457.098332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2457.122714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2457.147782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2457.173373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2457.201966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2457.235197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2457.941740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2458.152781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2458.169569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2458.194840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2458.224535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2458.248891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2458.271230] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.112@tcp [ 2458.286403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2458.305805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2458.332838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2458.348522] LNet: 119512:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2458.369806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2458.396972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2458.426869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2458.460945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2458.487966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2458.507980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2459.031986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.112@tcp [ 2459.258889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2459.289179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2459.335610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2459.376053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2459.421468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2459.478514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2459.532633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2459.569422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2459.589779] LNet: 119629:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2459.612855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2459.648993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2459.717791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2459.762484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2459.820182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2459.871966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2460.467438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2460.671383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2460.701792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2460.729713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2460.770842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2460.807726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2460.839897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2460.874672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2460.917477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2460.985772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2461.028260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2461.065257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2461.122554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2461.159510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2461.193913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2461.570119] LNet: 119792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2461.582298] LNetError: 119792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.609580] LNet: Removed LNI 192.168.201.12@tcp [ 2461.965800] Key type .llcrypt unregistered [ 2461.968072] Key type ._llcrypt unregistered [ 2480.892934] Key type ._llcrypt registered [ 2480.895966] Key type .llcrypt registered [ 2482.285452] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:50:31 (1780451431) [ 2483.084745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2483.097875] alg: No test for adler32 (adler32-zlib) [ 2484.020725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2484.341828] LNet: 120681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2484.878142] Key type .llcrypt unregistered [ 2484.880522] Key type ._llcrypt unregistered [ 2491.921505] Key type ._llcrypt registered [ 2491.924318] Key type .llcrypt registered [ 2492.188554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2492.221391] alg: No test for adler32 (adler32-zlib) [ 2493.334862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2493.369849] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2502.160023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.112@tcp [ 2513.002745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2513.035414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2513.070109] LNet: Added LNI 192.168.201.12@tcp1 [8/256/0/180] [ 2514.054489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2514.710993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.112@tcp [ 2514.907350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2514.912129] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.112@tcp [ 2514.948915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2514.984338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2515.006464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2515.037726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2515.061531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2515.084817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2515.111559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2515.134153] LNet: 121335:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2515.153530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2515.176647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2515.210275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2515.239726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2515.268724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2515.320383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2516.118417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2516.576857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2516.607640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2516.643173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2516.693816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2516.765516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2516.807289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2516.856636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2516.924150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2516.960441] LNet: 121452:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2516.980441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2517.045689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2517.099021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2517.146201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2517.213208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2517.246256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2517.962777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.112@tcp [ 2518.261556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2518.297792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2518.325271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2518.352302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2518.378660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2518.415562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2518.451738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2518.496862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2518.518764] LNet: 121570:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2518.542906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2518.572796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2518.603237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2518.633252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2518.660858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2518.686849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2519.444341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2519.714865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2519.740179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2519.761705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2519.791452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2519.821859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2519.857887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2519.893893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2519.917869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2519.942719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2519.967131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2519.993192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2520.019872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2520.042200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2520.081347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2526.232185] LNet: 121768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2526.267335] LNetError: 121768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2526.277161] LNet: Removed LNI 192.168.201.12@tcp [ 2526.305783] LNet: Removed LNI 192.168.201.12@tcp1 [ 2526.646733] Key type .llcrypt unregistered [ 2526.651221] Key type ._llcrypt unregistered [ 2549.451169] Key type ._llcrypt registered [ 2549.452719] Key type .llcrypt registered [ 2550.826344] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:51:39 (1780451499) [ 2559.334293] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:51:48 (1780451508) [ 2568.626429] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:51:57 (1780451517) [ 2578.558506] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:52:07 (1780451527) [ 2587.366286] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:52:16 (1780451536) [ 2595.247551] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:52:24 (1780451544) [ 2602.420892] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:52:31 (1780451551) [ 2610.130435] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:52:39 (1780451559) [ 2617.181859] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:52:46 (1780451566) [ 2624.596148] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:52:53 (1780451573) [ 2630.985348] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:53:00 (1780451580) [ 2639.047327] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:53:07 (1780451587) [ 2645.254675] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:53:14 (1780451594) [ 2653.332654] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:53:21 (1780451601) [ 2663.766869] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:53:32 (1780451612) [ 2676.519213] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:53:43 (1780451623) [ 2686.092132] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:53:55 (1780451635) [ 2693.741213] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:54:02 (1780451642) [ 2694.276274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2694.288333] alg: No test for adler32 (adler32-zlib) [ 2695.197790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2695.330212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2695.355655] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2696.620777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2696.638207] LNet: 132486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2696.644094] LNetError: 132486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2696.653208] LNet: Removed LNI 192.168.201.12@tcp [ 2696.664171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2696.706311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2696.734496] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2703.387851] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:54:12 (1780451652) [ 2711.185590] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:54:19 (1780451659) [ 2718.543882] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:54:27 (1780451667) [ 2725.917583] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:54:35 (1780451675) [ 2733.824368] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:54:42 (1780451682) [ 2740.685200] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:54:49 (1780451689) [ 2747.946113] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:54:57 (1780451697) [ 2756.123980] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:55:05 (1780451705) [ 2763.973297] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:55:13 (1780451713) [ 2770.409921] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:55:19 (1780451719) [ 2779.356232] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:55:27 (1780451727) [ 2789.527214] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:55:37 (1780451737) [ 2797.468290] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2505 sec ========== 21:55:46 (1780451746) [ 2798.085829] LNet: 139951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2798.093349] LNetError: 139951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2798.108350] LNet: Removed LNI 192.168.201.12@tcp [ 2798.886689] Key type .llcrypt unregistered [ 2798.888438] Key type ._llcrypt unregistered [ 2815.621519] Key type ._llcrypt registered [ 2815.626896] Key type .llcrypt registered [ 2815.991495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2816.000586] alg: No test for adler32 (adler32-zlib) [ 2817.228540] Lustre: Lustre: Build Version: 2.17.53_57_g315c119 [ 2817.476486] LNet: Added LNI 192.168.201.12@tcp [8/256/0/180] [ 2819.191235] Key type lgssc registered [ 2820.576781] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2889.483697] Lustre: Mounted lustre-client [ 2895.017331] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2903.566615] Lustre: Unmounted lustre-client [ 2932.930409] Key type lgssc unregistered [ 2933.255090] LNet: 142883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2933.271105] LNetError: 142883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2933.310358] LNet: Removed LNI 192.168.201.12@tcp [ 2934.203191] Key type .llcrypt unregistered [ 2934.207590] Key type ._llcrypt unregistered