[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 411926638 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002332] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008016] pid_max: default: 32768 minimum: 301 [ 0.009126] LSM: Security Framework initializing [ 0.011041] Yama: becoming mindful. [ 0.011963] SELinux: Initializing. [ 0.013009] *** VALIDATE selinux *** [ 0.020457] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025104] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026140] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029055] *** VALIDATE tmpfs *** [ 0.030336] *** VALIDATE proc *** [ 0.031233] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033243] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037016] Speculative Store Bypass: Vulnerable [ 0.040178] debug: unmapping init [mem 0xffffffffaf059000-0xffffffffaf060fff] [ 0.042170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043644] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048018] ... max period: 00007fffffffffff [ 0.049012] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051294] rcu: Hierarchical SRCU implementation. [ 0.053642] smp: Bringing up secondary CPUs ... [ 0.054599] x86: Booting SMP configuration: [ 0.055027] .... node #0, CPUs: #1 #2 #3 [ 0.059009] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.112019] node 0 deferred pages initialised in 47ms [ 0.114008] devtmpfs: initialized [ 0.115204] x86/mm: Memory block size: 128MB [ 0.117762] gcov: version magic: 0x41383552 [ 0.119221] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.120086] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.121302] pinctrl core: initialized pinctrl subsystem [ 0.122235] [ 0.122602] ************************************************************* [ 0.123014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124010] ** ** [ 0.125011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.126010] ** ** [ 0.127011] ** This means that this kernel is built to expose internal ** [ 0.128010] ** IOMMU data structures, which may compromise security on ** [ 0.129011] ** your system. ** [ 0.130009] ** ** [ 0.131013] ** If you see this message and you are not debugging the ** [ 0.132009] ** kernel, report this immediately to your vendor! ** [ 0.133008] ** ** [ 0.134010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135069] ************************************************************* [ 0.136751] NET: Registered protocol family 16 [ 0.137426] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.138056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.139052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140482] cpuidle: using governor menu [ 0.141862] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.144451] PCI: Using configuration type 1 for base access [ 0.146133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.154119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.155063] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.158064] cryptd: max_cpu_qlen set to 1000 [ 0.160245] ACPI: Added _OSI(Module Device) [ 0.162017] ACPI: Added _OSI(Processor Device) [ 0.164013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.165011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.168908] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.176112] ACPI: Interpreter enabled [ 0.176957] ACPI: PM: (supports S0 S3 S4 S5) [ 0.177012] ACPI: Using IOAPIC for interrupt routing [ 0.179102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.181356] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.191562] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.194039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.197017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.199073] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.203261] acpiphp: Slot [2] registered [ 0.204173] acpiphp: Slot [5] registered [ 0.205118] acpiphp: Slot [6] registered [ 0.207116] acpiphp: Slot [3] registered [ 0.208082] acpiphp: Slot [4] registered [ 0.209082] acpiphp: Slot [7] registered [ 0.210108] acpiphp: Slot [8] registered [ 0.212067] acpiphp: Slot [9] registered [ 0.213031] acpiphp: Slot [10] registered [ 0.214114] acpiphp: Slot [11] registered [ 0.216097] acpiphp: Slot [12] registered [ 0.217093] acpiphp: Slot [13] registered [ 0.219189] acpiphp: Slot [14] registered [ 0.220097] acpiphp: Slot [15] registered [ 0.222162] acpiphp: Slot [16] registered [ 0.223126] acpiphp: Slot [17] registered [ 0.224131] acpiphp: Slot [18] registered [ 0.226075] acpiphp: Slot [19] registered [ 0.227103] acpiphp: Slot [20] registered [ 0.228084] acpiphp: Slot [21] registered [ 0.229072] acpiphp: Slot [22] registered [ 0.230081] acpiphp: Slot [23] registered [ 0.231071] acpiphp: Slot [24] registered [ 0.232103] acpiphp: Slot [25] registered [ 0.234074] acpiphp: Slot [26] registered [ 0.235076] acpiphp: Slot [27] registered [ 0.236078] acpiphp: Slot [28] registered [ 0.237091] acpiphp: Slot [29] registered [ 0.238092] acpiphp: Slot [30] registered [ 0.239067] acpiphp: Slot [31] registered [ 0.240046] PCI host bridge to bus 0000:00 [ 0.242018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.243014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.245029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.247016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.249015] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.250029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.251000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.251000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.254534] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.262009] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.267040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.269014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.271021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.273018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.275571] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.277738] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.281048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.284393] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.287967] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.298013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.301015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.306033] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.313021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.319024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.338019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.353252] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.366019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.380030] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.396020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.406368] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.408336] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.410337] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.412297] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.414152] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.417124] iommu: Default domain type: Passthrough [ 0.418432] SCSI subsystem initialized [ 0.419173] ACPI: bus type USB registered [ 0.421102] usbcore: registered new interface driver usbfs [ 0.422096] usbcore: registered new interface driver hub [ 0.424075] usbcore: registered new device driver usb [ 0.425155] pps_core: LinuxPPS API ver. 1 registered [ 0.426010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.429061] PTP clock support registered [ 0.430137] EDAC MC: Ver: 3.0.0 [ 0.431151] PCI: Using ACPI for IRQ routing [ 0.432000] NetLabel: Initializing [ 0.432000] NetLabel: domain hash size = 128 [ 0.433009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.435072] NetLabel: unlabeled traffic allowed by default [ 0.437124] vgaarb: loaded [ 0.438390] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.439008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.445138] clocksource: Switched to clocksource kvm-clock [ 0.569485] VFS: Disk quotas dquot_6.6.0 [ 0.571880] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.574483] *** VALIDATE ramfs *** [ 0.575873] *** VALIDATE hugetlbfs *** [ 0.578202] pnp: PnP ACPI init [ 0.580737] pnp: PnP ACPI: found 6 devices [ 0.596842] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.599395] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.600932] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.602989] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.604772] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.606754] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.609649] NET: Registered protocol family 2 [ 0.611915] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.616719] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.619713] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.624634] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.627141] TCP: Hash tables configured (established 65536 bind 65536) [ 0.630037] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.632113] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.633747] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.635806] NET: Registered protocol family 1 [ 0.638031] RPC: Registered named UNIX socket transport module. [ 0.640334] RPC: Registered udp transport module. [ 0.642199] RPC: Registered tcp transport module. [ 0.644183] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.646265] NET: Registered protocol family 44 [ 0.648409] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.649764] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.651303] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.653987] PCI: CLS 0 bytes, default 64 [ 0.655631] Unpacking initramfs... [ 2.105218] debug: unmapping init [mem 0xffff8ec6fcc64000-0xffff8ec6fffcffff] [ 2.108897] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.110594] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.112711] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.648692] Initialise system trusted keyrings [ 2.650153] Key type blacklist registered [ 2.651731] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.660721] zbud: loaded [ 2.663676] *** VALIDATE nfs *** [ 2.665086] *** VALIDATE nfs4 *** [ 2.666920] pstore: using deflate compression [ 2.671081] Platform Keyring initialized [ 2.787189] NET: Registered protocol family 38 [ 2.789179] Key type asymmetric registered [ 2.791018] Asymmetric key parser 'x509' registered [ 2.793161] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.796248] io scheduler mq-deadline registered [ 2.798112] io scheduler kyber registered [ 2.802232] io scheduler bfq registered [ 2.804421] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.808155] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.811560] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.814732] ACPI: Power Button [PWRF] [ 2.820723] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.828130] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.839125] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.867453] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.895505] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.899969] Non-volatile memory driver v1.3 [ 2.901718] Linux agpgart interface v0.103 [ 2.935148] virtio_blk virtio1: [vda] 139368 512-byte logical blocks (71.4 MB/68.1 MiB) [ 2.938452] vda: detected capacity change from 0 to 71356416 [ 2.958687] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.961164] vdb: detected capacity change from 0 to 1073741824 [ 2.976908] libphy: Fixed MDIO Bus: probed [ 2.987167] usbcore: registered new interface driver usbserial_generic [ 2.989018] usbserial: USB Serial support registered for generic [ 2.990851] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.995442] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.997173] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.999524] mousedev: PS/2 mouse device common for all mice [ 3.002628] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.004554] rtc_cmos 00:05: RTC can wake from S4 [ 3.008843] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.009112] rtc_cmos 00:05: registered as rtc0 [ 3.014357] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.014690] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.020174] intel_pstate: CPU model not supported [ 3.023534] hid: raw HID events driver (C) Jiri Kosina [ 3.025833] usbcore: registered new interface driver usbhid [ 3.027925] usbhid: USB HID core driver [ 3.029605] drop_monitor: Initializing network drop monitor service [ 3.032119] Initializing XFRM netlink socket [ 3.034421] NET: Registered protocol family 10 [ 3.037419] Segment Routing with IPv6 [ 3.038868] NET: Registered protocol family 17 [ 3.040965] mpls_gso: MPLS GSO support [ 3.046610] RAS: Correctable Errors collector initialized. [ 3.048826] AVX version of gcm_enc/dec engaged. [ 3.050625] AES CTR mode by8 optimization enabled [ 3.137763] sched_clock: Marking stable (3137727528, 0)->(4008606264, -870878736) [ 3.141409] registered taskstats version 1 [ 3.143667] Loading compiled-in X.509 certificates [ 3.146044] zswap: loaded using pool lzo/zbud [ 3.173135] Key type big_key registered [ 3.185191] Key type encrypted registered [ 3.186975] ima: No TPM chip found, activating TPM-bypass! [ 3.189152] ima: Allocated hash algorithm: sha1 [ 3.191085] ima: No architecture policies found [ 3.193201] evm: Initialising EVM extended attributes: [ 3.195057] evm: security.selinux [ 3.196395] evm: security.ima [ 3.197514] evm: security.capability [ 3.198915] evm: HMAC attrs: 0x1 [ 3.201335] rtc_cmos 00:05: setting system clock to 2026-02-10 00:33:19 UTC (1770683599) [ 3.207703] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.211105] debug: unmapping init [mem 0xffffffffaed82000-0xffffffffaf058fff] [ 3.223299] Write protecting the kernel read-only data: 28672k [ 3.226856] debug: unmapping init [mem 0xffffffffad403000-0xffffffffad5fffff] [ 3.230189] debug: unmapping init [mem 0xffffffffadd14000-0xffffffffaddfffff] [ 3.267702] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.276830] systemd[1]: Detected virtualization kvm. [ 3.278838] systemd[1]: Detected architecture x86-64. [ 3.281123] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.311475] systemd[1]: No hostname configured. [ 3.313323] systemd[1]: Set hostname to . [ 3.315494] random: systemd: uninitialized urandom read (16 bytes read) [ 3.318447] systemd[1]: Initializing machine ID from random generator. [ 3.389556] random: ln: uninitialized urandom read (6 bytes read) [ 3.503413] random: systemd: uninitialized urandom read (16 bytes read) [ 3.506879] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.513561] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.520257] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.206970] device-mapper: uevent: version 1.0.3 [ 4.209082] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.006485] virtio_net virtio0 ens2: renamed from eth0 [ 5.152507] scsi host0: ata_piix [ 5.227415] scsi host1: ata_piix [ 5.228800] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.230763] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.553909] dracut-initqueue[591]: RTNETLINK answers: File exists [ 9.704275] random: crng init done [ 9.705564] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.228457] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.442230] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.677696] SELinux: Disabled at runtime. [ 11.738268] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.748806] systemd[1]: Detected virtualization kvm. [ 11.750618] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.213924] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.216391] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.219035] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.221453] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.223330] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.230445] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.238538] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ 12.259830] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.627888] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.884942] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.900385] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.997681] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.014134] EDAC sbridge: Ver: 1.1.2 [ 14.010511] Key type dns_resolver registered [ 14.315851] NFS: Registering the id_resolver key type [ 14.317880] Key type id_resolver registered [ 14.319532] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg410-client login: [ 88.174831] libcfs: loading out-of-tree module taints kernel. [ 88.547249] Key type ._llcrypt registered [ 88.548874] Key type .llcrypt registered [ 89.185315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 89.195930] alg: No test for adler32 (adler32-zlib) [ 90.757487] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 91.661383] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 93.551253] Key type lgssc registered [ 96.232901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.066074] hrtimer: interrupt took 3424403 ns [ 248.913647] Lustre: Mounted lustre-client [ 255.588090] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 274.408166] Lustre: lustre-OST0000-osc-ffff8ec747f3f000: disconnect after 22s idle [ 279.279912] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing check_logdir /tmp/testlogs/ [ 285.286808] Lustre: DEBUG MARKER: oleg410-client.virtnet: executing yml_node [ 291.567579] Lustre: DEBUG MARKER: Client: 2.17.50.65 [ 295.004808] Lustre: DEBUG MARKER: MDS: 2.17.50.65 [ 298.427729] Lustre: DEBUG MARKER: OSS: 2.17.50.65 [ 300.551504] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Feb 9 19:38:15 EST 2026 [ 321.257641] LustreError: 7919:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ec747f3f000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 321.272912] LustreError: 7919:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 321.305352] Lustre: Unmounted lustre-client [ 343.146931] Key type lgssc unregistered [ 343.501441] LNet: 8365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.511341] LNetError: 8365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.563041] LNet: Removed LNI 192.168.204.10@tcp [ 345.561280] Key type .llcrypt unregistered [ 345.562910] Key type ._llcrypt unregistered [ 376.033176] Key type ._llcrypt registered [ 376.038278] Key type .llcrypt registered [ 376.376918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 376.395996] alg: No test for adler32 (adler32-zlib) [ 377.356152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 377.450329] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 377.480532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 382.106397] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 382.476122] LNet: 9243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.493948] LNetError: 9243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 382.535683] LNet: Removed LNI 192.168.204.10@tcp [ 383.078371] Key type .llcrypt unregistered [ 383.080956] Key type ._llcrypt unregistered [ 391.238319] Key type ._llcrypt registered [ 391.241190] Key type .llcrypt registered [ 393.243914] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:39:47 (1770683987) [ 393.770626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 393.795202] alg: No test for adler32 (adler32-zlib) [ 394.874353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.993955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.006844] LNet: 9650:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.017698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.044716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 395.054667] LNet: 9657:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 404.019656] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:39:58 (1770683998) [ 404.163444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 404.202913] LNet: 10228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 404.251104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 404.415977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 413.118518] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:40:07 (1770684007) [ 413.333832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.392756] LNet: 10821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.447092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.610896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 422.019834] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:40:16 (1770684016) [ 422.125690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.166942] LNet: 11399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.207911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.367593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 429.906342] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:40:24 (1770684024) [ 430.395909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.427467] LNet: 11991:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 430.452028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.544646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 431.176091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 439.642362] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:40:34 (1770684034) [ 440.058374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.100764] LNet: 12582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.137893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.273580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 441.189073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 442.078194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 451.153445] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:40:45 (1770684045) [ 451.282974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.348279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.493737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 461.563753] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:40:55 (1770684055) [ 461.689410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.732108] LNet: 13759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.736963] LNet: 13759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 461.770469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.885690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 461.970775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 462.548839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 462.607716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 463.407207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 463.494257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 464.054136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 464.112889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 464.748924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 464.809285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 465.382818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 465.439886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 466.009856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 466.056897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 466.414618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 466.442547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 466.741563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 466.785633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 467.188644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 467.226877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 467.603670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 467.648818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 468.147497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 468.204518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 468.693091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 468.752563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 475.016761] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:41:10 (1770684070) [ 475.329773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.396583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.482473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 475.510948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 482.910860] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:41:17 (1770684077) [ 483.150644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.246339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.320884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 483.368486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 490.707426] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:41:25 (1770684085) [ 490.898565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.989232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.160587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 491.218773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 499.847512] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:41:34 (1770684094) [ 500.039988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.096882] LNet: 16200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.105930] LNet: 16200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 500.157212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.309153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 500.353644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 508.119136] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 19:41:42 (1770684102) [ 508.286226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.406945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.556240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 508.666779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 516.677082] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 19:41:51 (1770684111) [ 516.815942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.896196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.043951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 517.085647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 525.447793] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 19:41:59 (1770684119) [ 525.539346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.626379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.712607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 526.077757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 533.084833] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 19:42:07 (1770684127) [ 533.526382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.557938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.739348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 533.956980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 542.252342] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 19:42:17 (1770684137) [ 542.457655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.482470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.620693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.839916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 550.154794] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 19:42:25 (1770684145) [ 550.552520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.579692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.995433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 551.215735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 558.611925] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 19:42:33 (1770684153) [ 558.888979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.952148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.036304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 559.107349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 566.235692] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 19:42:41 (1770684161) [ 566.546974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.579021] LNet: 21040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.582989] LNet: 21040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 566.607521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.710391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 566.784660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 574.555949] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 19:42:49 (1770684169) [ 574.948829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 575.033516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 575.156407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 583.110671] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 19:42:57 (1770684177) [ 583.258586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.361164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.529756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 591.079649] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 19:43:05 (1770684185) [ 591.337776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.372258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.436886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 591.854386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 600.233224] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 19:43:14 (1770684194) [ 600.367448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.445854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.600464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 600.733505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 609.482586] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 19:43:23 (1770684203) [ 609.611415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.696394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.897260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 609.962614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 617.276391] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 19:43:32 (1770684212) [ 617.374085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.441624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.584468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 617.663648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 625.431993] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 19:43:40 (1770684220) [ 625.689695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 625.784195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.862173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 625.876265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 632.517853] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 19:43:47 (1770684227) [ 632.860604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.908835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.964297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 633.037725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 633.062135] LNetError: 25765:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 633.120527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 633.165565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 640.679400] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 19:43:55 (1770684235) [ 640.799751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 640.846622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.943372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jCokXAkhSV [ 641.010746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jCokXAkhSV [ 641.054959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jCokXAkhSV [ 651.014105] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 19:44:04 (1770684244) [ 651.507934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.554454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.625324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 651.661372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 651.713137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 662.061333] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 19:44:16 (1770684256) [ 662.465219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.510045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.565610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 665.543509] LNet: Message forwarding will be enabled in 2s [ 668.640529] LNet: Message forwarding enabled [ 669.175889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 669.218952] LNet: Message forwarding will stop in 2s to 6s [ 672.869143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 674.366513] LNet: Message forwarding disabled [ 676.210833] LNet: Message forwarding will be enabled in 2s [ 679.521127] LNet: Message forwarding enabled [ 684.414860] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 19:44:39 (1770684279) [ 684.874383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.946605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.098454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 694.764378] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 19:44:49 (1770684289) [ 694.959562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.004761] LNet: 28782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.018392] LNet: 28782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 695.076817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.167464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 695.200614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 695.232564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 695.262809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 695.283290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 695.301870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 695.324346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 695.351383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 695.378925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 695.413671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 695.445397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 695.482689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 695.518130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 695.566519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 695.611047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 695.665388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 695.711371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 695.764672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 695.808579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 695.836534] LNetError: 28829:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 702.816254] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 19:44:57 (1770684297) [ 703.120261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.193532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.259989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 710.811480] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 19:45:05 (1770684305) [ 712.989357] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg410-server [ 714.863915] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 19:45:09 (1770684309) [ 715.095828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.129920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.177699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.206473] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 716.678941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[11-15]@tcp [ 716.697411] LNet: 30546:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 716.759921] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.14@tcp is being used as a gateway but routing feature is not turned on [ 716.780308] LNetError: 30336:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.14@tcp has gone from up to down [ 717.764637] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 717.773618] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 718.829974] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 718.847497] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 720.880313] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.11@tcp is being used as a gateway but routing feature is not turned on [ 720.915684] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 724.029146] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 19:45:19 (1770684319) [ 724.966853] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 724.983966] LNetError: 30336:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 726.452461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg410-server [ 728.213283] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 19:45:23 (1770684323) [ 728.397835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.448926] LNetError: 31479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.477754] LNet: Removed LNI 192.168.204.10@tcp [ 728.522953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.576137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 728.600205] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 729.422510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[11-16/2]@tcp [ 729.455945] LNet: 31599:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 729.473903] LNet: 31599:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 729.501821] LNetError: 31486:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.15@tcp has gone from up to down [ 729.507561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[11-16/2]@tcp [ 729.511350] LNetError: 31486:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 739.613209] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 19:45:32 (1770684332) [ 739.750769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 739.773230] LNetError: 32184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.807973] LNet: Removed LNI 192.168.204.10@tcp [ 739.877827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.059458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 740.149462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.192687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.293809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 740.386982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.457599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.567859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 740.684931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 740.726606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.815603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 740.977485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.066343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.130428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 741.240287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.271320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.321587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 741.397962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.433365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.521605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 741.630937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.681911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.775509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 741.937646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.985772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.111158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 742.206877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.268922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.371661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 742.525692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.582768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.653877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 742.714797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.754832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.795834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 750.752846] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 19:45:45 (1770684345) [ 753.623751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg410-server [ 756.095943] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 19:45:50 (1770684350) [ 757.873802] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg410-server [ 760.389654] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 19:45:54 (1770684354) [ 761.916808] Key type .llcrypt unregistered [ 761.920499] Key type ._llcrypt unregistered [ 770.587569] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 770.693237] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 770.696358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 770.699859] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 773.521920] Key type ._llcrypt registered [ 773.523391] Key type .llcrypt registered [ 773.656037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 773.688745] alg: No test for adler32 (adler32-zlib) [ 774.732424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.800703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 774.821119] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 775.749899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 775.780529] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.163659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 778.186864] LNet: Removed LNI 10.1.2.3@tcp [ 778.343592] LNet: 34314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 778.352572] LNetError: 34314:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 778.369974] LNet: Removed LNI 192.168.204.10@tcp [ 779.274481] Key type .llcrypt unregistered [ 779.281770] Key type ._llcrypt unregistered [ 794.555258] Key type ._llcrypt registered [ 794.557598] Key type .llcrypt registered [ 796.161043] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 19:46:30 (1770684390) [ 797.063386] Key type .llcrypt unregistered [ 797.065091] Key type ._llcrypt unregistered [ 804.649025] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 804.893265] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 804.905493] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 804.915254] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.818747] Key type ._llcrypt registered [ 807.821309] Key type .llcrypt registered [ 808.030482] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.045501] alg: No test for adler32 (adler32-zlib) [ 809.087247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.169965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.218565] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 810.334847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 810.365990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 813.225247] LNet: 35800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 813.240520] LNetError: 35800:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 813.243215] LNet: Removed LNI 192.168.204.10@tcp [ 813.262290] LNet: Removed LNI 10.1.2.3@tcp [ 813.951736] Key type .llcrypt unregistered [ 813.957468] Key type ._llcrypt unregistered [ 827.484637] Key type ._llcrypt registered [ 827.487508] Key type .llcrypt registered [ 828.856856] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 19:47:03 (1770684423) [ 829.239301] Key type .llcrypt unregistered [ 829.243606] Key type ._llcrypt unregistered [ 836.378779] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 836.504489] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 836.532141] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 836.554549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 839.458642] Key type ._llcrypt registered [ 839.460824] Key type .llcrypt registered [ 839.628446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 839.638530] alg: No test for adler32 (adler32-zlib) [ 840.597035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.643841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 840.670303] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 842.920771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 842.960861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 845.649809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 845.667577] LNet: Removed LNI 10.1.2.3@tcp [ 845.682648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 845.709253] LNet: Removed LNI 10.1.2.31@tcp [ 845.725154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 845.746785] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 846.853651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 846.890414] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 848.275246] LNet: 37715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.293448] LNetError: 37715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 848.303533] LNet: Removed LNI 10.1.2.3@tcp [ 848.338244] LNet: Removed LNI 10.1.2.31@tcp [ 849.012434] Key type .llcrypt unregistered [ 849.019659] Key type ._llcrypt unregistered [ 862.751419] Key type ._llcrypt registered [ 862.759350] Key type .llcrypt registered [ 864.183584] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:47:38 (1770684458) [ 864.865890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 864.886924] alg: No test for adler32 (adler32-zlib) [ 865.936161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.069581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.100241] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 867.007365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.025451] LNet: 38664:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.032846] LNetError: 38664:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 867.047491] LNet: Removed LNI 192.168.204.10@tcp [ 867.070832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.129172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.10@tcp [ 867.174529] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 868.230190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.10@tcp [ 868.281746] LNet: Removed LNI 192.168.204.10@tcp [ 868.391413] LNet: 38799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.189479] Key type .llcrypt unregistered [ 869.190827] Key type ._llcrypt unregistered [ 883.324503] Key type ._llcrypt registered [ 883.331226] Key type .llcrypt registered [ 885.506958] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:47:59 (1770684479) [ 888.360087] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg410-server [ 891.077491] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:48:05 (1770684485) [ 892.166604] Key type .llcrypt unregistered [ 892.173366] Key type ._llcrypt unregistered [ 901.236230] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.581866] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.597726] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.611156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.592775] Key type ._llcrypt registered [ 904.594289] Key type .llcrypt registered [ 904.836139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.860776] alg: No test for adler32 (adler32-zlib) [ 905.837752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 905.902120] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 906.682374] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 908.840455] LNet: 40633:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.845581] LNetError: 40633:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.865171] LNet: Removed LNI 192.168.204.10@tcp1 [ 908.887735] LNet: Removed LNI 10.1.2.3@tcp2 [ 909.621631] Key type .llcrypt unregistered [ 909.623585] Key type ._llcrypt unregistered [ 924.187721] Key type ._llcrypt registered [ 924.188933] Key type .llcrypt registered [ 926.026649] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:48:40 (1770684520) [ 926.423198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.435892] alg: No test for adler32 (adler32-zlib) [ 927.490480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.613655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.634527] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 928.902051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.944526] LNet: 41598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.962780] LNetError: 41598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.979713] LNet: Removed LNI 192.168.204.10@tcp [ 929.024505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.154851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 929.211878] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 939.276561] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:48:53 (1770684533) [ 939.411948] LNet: 42181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.426757] LNetError: 42181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.490931] LNet: Removed LNI 192.168.204.10@tcp [ 940.448145] Key type .llcrypt unregistered [ 940.456284] Key type ._llcrypt unregistered [ 948.279703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 948.447895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 948.453666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 948.466335] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.256510] Key type ._llcrypt registered [ 951.260582] Key type .llcrypt registered [ 951.580569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 951.604441] alg: No test for adler32 (adler32-zlib) [ 952.636768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.716771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 952.749899] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 954.004509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 954.028924] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.357294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.393678] LNet: 42924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.401584] LNetError: 42924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.405815] LNet: Removed LNI 192.168.204.10@tcp [ 957.416130] LNet: Removed LNI 10.1.2.3@tcp [ 957.469109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.546763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 957.593086] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 957.610395] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 958.354922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.384296] LNet: 42951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.390550] LNetError: 42951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.395900] LNetError: 42951:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 958.400713] LNet: Removed LNI 192.168.204.10@tcp [ 958.409941] LNet: Removed LNI 10.1.2.3@tcp [ 958.438075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.575590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 958.637287] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 958.647834] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 959.073966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 959.093656] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 960.294924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 960.311850] LNet: 43114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 960.316598] LNetError: 43114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.320756] LNetError: 43114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 960.325429] LNet: Removed LNI 192.168.204.10@tcp [ 960.328289] LNet: Removed LNI 10.1.2.3@tcp [ 960.338174] LNet: Removed LNI 192.168.204.10@tcp2 [ 960.372582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.442442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 960.465474] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 960.470131] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 960.474982] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 970.227402] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:49:25 (1770684565) [ 970.472757] LNet: 43712:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.479629] LNetError: 43712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.484854] LNetError: 43712:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 970.490783] LNet: Removed LNI 192.168.204.10@tcp [ 970.494219] LNet: Removed LNI 10.1.2.3@tcp [ 970.508977] LNet: Removed LNI 192.168.204.10@tcp2 [ 971.157859] Key type .llcrypt unregistered [ 971.161996] Key type ._llcrypt unregistered [ 977.839722] Key type ._llcrypt registered [ 977.843233] Key type .llcrypt registered [ 978.152813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 978.189497] alg: No test for adler32 (adler32-zlib) [ 979.162811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.226853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 979.261633] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 980.100337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.117330] LNet: 44102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.127246] LNetError: 44102:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.148219] LNet: Removed LNI 192.168.204.10@tcp [ 980.167614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.255692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 980.293939] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 980.646912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.661083] LNet: 44132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.666980] LNetError: 44132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.677350] LNet: Removed LNI 192.168.204.10@tcp [ 980.698099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.748448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 980.766600] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 981.125177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.161208] LNet: Removed LNI 192.168.204.10@tcp [ 981.180534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.237258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 981.263698] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 981.614215] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.811126] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 981.827318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 981.832522] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.270483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.320191] LNet: 44240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.323364] LNet: 44240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 984.346311] LNetError: 44240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.354859] LNetError: 44240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 984.374660] LNet: Removed LNI 192.168.204.10@tcp [ 984.405573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.501134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 984.523868] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 992.170829] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:49:46 (1770684586) [ 992.433489] LNet: 44835:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.452833] LNetError: 44835:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.531962] LNet: Removed LNI 192.168.204.10@tcp [ 993.168248] Key type .llcrypt unregistered [ 993.170561] Key type ._llcrypt unregistered [ 999.735957] Key type ._llcrypt registered [ 999.740448] Key type .llcrypt registered [ 1000.076409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.088220] alg: No test for adler32 (adler32-zlib) [ 1000.949890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1000.991912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1001.009193] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1001.751651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1001.764447] LNet: 45233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.770522] LNetError: 45233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1001.778080] LNet: Removed LNI 192.168.204.10@tcp [ 1001.797351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1001.844642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1001.860389] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1010.332315] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:50:04 (1770684604) [ 1010.510298] LNet: 45821:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.516379] LNetError: 45821:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.555649] LNet: Removed LNI 192.168.204.10@tcp [ 1011.339875] Key type .llcrypt unregistered [ 1011.342299] Key type ._llcrypt unregistered [ 1018.444971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.688120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1018.701354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1018.705524] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.559376] Key type ._llcrypt registered [ 1021.564330] Key type .llcrypt registered [ 1021.879964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.923702] alg: No test for adler32 (adler32-zlib) [ 1023.054627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.160573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1023.199325] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1024.482503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1024.510814] LNet: 46273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.522293] LNetError: 46273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.549269] LNet: Removed LNI 192.168.204.10@tcp [ 1024.600209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.708867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1024.774587] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1033.474486] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:50:27 (1770684627) [ 1033.549535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1033.585253] LNet: 46864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1033.592377] LNetError: 46864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.614524] LNet: Removed LNI 192.168.204.10@tcp [ 1033.684499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.836121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1041.838838] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:50:36 (1770684636) [ 1042.104770] LNet: 47452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.931783] Key type .llcrypt unregistered [ 1042.937224] Key type ._llcrypt unregistered [ 1049.907452] Key type ._llcrypt registered [ 1049.912347] Key type .llcrypt registered [ 1050.284693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.306706] alg: No test for adler32 (adler32-zlib) [ 1051.413975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1051.530962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1051.567206] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1053.163922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1053.228457] LNet: 47849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1053.252266] LNetError: 47849:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1053.270213] LNet: Removed LNI 192.168.204.10@tcp [ 1053.341748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.449689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1053.504109] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1062.171703] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:50:57 (1770684657) [ 1062.727433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1062.771550] LNet: 48442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.790715] LNetError: 48442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.814733] LNet: Removed LNI 192.168.204.10@tcp [ 1062.849728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.956939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1071.034074] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:51:05 (1770684665) [ 1072.767133] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1075.173598] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:51:09 (1770684669) [ 1077.725263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1079.749534] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:51:14 (1770684674) [ 1081.543958] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1083.841895] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:51:18 (1770684678) [ 1085.843960] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1087.618426] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:51:22 (1770684682) [ 1089.286866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1091.348318] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:51:26 (1770684686) [ 1091.774335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1091.813055] LNet: 50497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.838399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1091.923776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1091.949243] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1092.616322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.10 [ 1092.626410] LNet: 50630:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1092.633025] LNet: Removed LNI 192.168.204.10@tcp [ 1099.617259] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:51:34 (1770684694) [ 1099.831642] LNet: 51194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1099.838710] LNetError: 51194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1099.871178] LNet: Removed LNI 192.168.204.10@tcp [ 1100.634972] Key type .llcrypt unregistered [ 1100.643045] Key type ._llcrypt unregistered [ 1108.185567] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1108.519703] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1108.524925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1108.531203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1111.699285] Key type ._llcrypt registered [ 1111.700714] Key type .llcrypt registered [ 1111.983903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.001938] alg: No test for adler32 (adler32-zlib) [ 1113.592840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1119.921154] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:51:55 (1770684715) [ 1120.330266] LNet: 52105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.344069] LNetError: 52105:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1120.380290] LNet: Removed LNI 10.1.2.3@tcp [ 1121.289361] Key type .llcrypt unregistered [ 1121.297942] Key type ._llcrypt unregistered [ 1129.441232] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1129.697485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1129.701700] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1129.710990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1132.439128] Key type ._llcrypt registered [ 1132.442786] Key type .llcrypt registered [ 1132.690661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1132.749625] alg: No test for adler32 (adler32-zlib) [ 1133.827607] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1141.521444] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:52:16 (1770684736) [ 1141.932878] LNet: 52995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1141.944212] LNetError: 52995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1141.957382] LNet: Removed LNI 10.1.2.3@tcp [ 1142.884825] Key type .llcrypt unregistered [ 1142.891484] Key type ._llcrypt unregistered [ 1151.390217] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1151.680213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1151.682801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1151.691942] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1154.595765] Key type ._llcrypt registered [ 1154.600494] Key type .llcrypt registered [ 1154.913678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1154.933696] alg: No test for adler32 (adler32-zlib) [ 1156.061285] LNet: 53293:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1156.082951] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1165.469337] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:52:39 (1770684759) [ 1165.608226] LNet: 53877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.616672] LNetError: 53877:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1165.668252] LNet: Removed LNI 10.1.2.3@tcp [ 1166.999578] Key type .llcrypt unregistered [ 1167.001395] Key type ._llcrypt unregistered [ 1174.820730] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1175.207089] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1175.215485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1175.231034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1176.750128] Key type ._llcrypt registered [ 1176.751785] Key type .llcrypt registered [ 1176.954334] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1176.994733] alg: No test for adler32 (adler32-zlib) [ 1177.969763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.054859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1186.780469] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:53:01 (1770684781) [ 1187.335446] LNet: 55104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1187.344846] LNetError: 55104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1187.360195] LNet: Removed LNI 10.1.2.3@tcp [ 1188.131556] Key type .llcrypt unregistered [ 1188.135620] Key type ._llcrypt unregistered [ 1194.978117] Key type ._llcrypt registered [ 1194.979583] Key type .llcrypt registered [ 1195.378386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1195.389945] alg: No test for adler32 (adler32-zlib) [ 1196.365973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1196.412856] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1206.055851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1207.127686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_interrupt -r 1 [ 1207.224386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_interrupt -r 1 [ 1207.298958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1207.331487] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1207.643957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1207.675663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1207.711472] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1207.712040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1207.773559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1207.820706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1207.860831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1207.905889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1207.955880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1207.984890] LNet: 55538:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1208.021879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1208.072441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1208.122297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.167804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1208.209028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1208.243517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1208.291511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1209.021439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_dropped -r 1 [ 1209.067962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_dropped -r 1 [ 1209.135546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1209.158262] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1209.448988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1209.496459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1209.563766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1209.618562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.680945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.723636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.797858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.855035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.893328] LNet: 55647:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.922952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.009644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.089247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.116366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.151832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.185289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.218301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.865888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_aborted -r 1 [ 1210.941023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_aborted -r 1 [ 1210.991279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1211.062841] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1211.381568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.425449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1211.458877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.522723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.580204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.636423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.679412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.719306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.734752] LNet: 55757:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.751561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.786825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.816150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.850264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.894287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.942166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.980872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1212.529451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_no_route -r 1 [ 1212.567323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_no_route -r 1 [ 1212.620946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1212.911897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.948513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1213.006901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.047203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.079257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.124135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.185691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.231974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.320964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.379841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.437196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.485939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.545227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.582453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.628269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.300681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_timeout -r 1 [ 1214.347075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_timeout -r 1 [ 1214.404134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1214.431242] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1214.438568] LNet: Skipped 2 previous similar messages [ 1214.681908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.734351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.757812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.809665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.841838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.885373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.954197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1215.011993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1215.031411] LNet: 55975:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1215.035387] LNet: 55975:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1215.051395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1215.086394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1215.147281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.195793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.236916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.278208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.342764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1216.106561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_error -r 1 [ 1216.147005] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_error -r 1 [ 1216.199662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1216.525646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.558724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1216.588830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.638049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.673987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.727765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.774292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.821885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.860637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.893917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.931679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.985663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.061801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.116041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.198763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1217.589779] LNet: 56125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.603158] LNetError: 56125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.624582] LNet: Removed LNI 192.168.204.10@tcp [ 1217.983769] Key type .llcrypt unregistered [ 1217.985480] Key type ._llcrypt unregistered [ 1238.019379] Key type ._llcrypt registered [ 1238.028582] Key type .llcrypt registered [ 1239.503711] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:53:54 (1770684834) [ 1240.557287] Key type .llcrypt unregistered [ 1240.560317] Key type ._llcrypt unregistered [ 1248.624179] Key type ._llcrypt registered [ 1248.627471] Key type .llcrypt registered [ 1249.082260] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.140196] alg: No test for adler32 (adler32-zlib) [ 1250.158298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1250.205349] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1259.118526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1271.158312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.221687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1271.255216] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1273.318482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_interrupt -r 1 [ 1273.348344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_interrupt -r 1 [ 1273.384843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_interrupt -r 1 [ 1273.416956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_interrupt -r 1 [ 1273.445529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_interrupt -r 1 [ 1273.470687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_interrupt -r 1 [ 1273.498769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_interrupt -r 1 [ 1273.527712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_interrupt -r 1 [ 1273.563469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1273.600334] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1273.606859] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1273.918517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1273.952268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1274.000513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1274.043959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1274.077925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1274.145237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1274.180590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1274.211178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1274.224121] LNet: 57589:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1274.237886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1274.281304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1274.333117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1274.386574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1274.436313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1274.506611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1274.552894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1275.254786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_dropped -r 1 [ 1275.301732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_dropped -r 1 [ 1275.351043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_dropped -r 1 [ 1275.392666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_dropped -r 1 [ 1275.431922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_dropped -r 1 [ 1275.466796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_dropped -r 1 [ 1275.505511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_dropped -r 1 [ 1275.542720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_dropped -r 1 [ 1275.639754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1275.697744] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1275.711401] LNet: Skipped 2 previous similar messages [ 1275.965991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.992965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1276.022261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1276.048102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1276.100084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1276.140966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1276.186630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1276.230658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1276.258707] LNet: 57710:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1276.301275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1276.379458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1276.410804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1276.458073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1276.512863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.576165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1276.617079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1277.373759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_aborted -r 1 [ 1277.437626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_aborted -r 1 [ 1277.481366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_aborted -r 1 [ 1277.507946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_aborted -r 1 [ 1277.539138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_aborted -r 1 [ 1277.570345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_aborted -r 1 [ 1277.606725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_aborted -r 1 [ 1277.631281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_aborted -r 1 [ 1277.658641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1277.681806] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1277.691761] LNet: Skipped 2 previous similar messages [ 1277.982493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1278.019977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1278.066215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1278.110190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1278.138745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1278.157621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1278.172444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1278.194696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1278.227682] LNet: 57832:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1278.260263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1278.284953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1278.325236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.354978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.384251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.423098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.465261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1279.283907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_no_route -r 1 [ 1279.366615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_no_route -r 1 [ 1279.472765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_no_route -r 1 [ 1279.537551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_no_route -r 1 [ 1279.582858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_no_route -r 1 [ 1279.646925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_no_route -r 1 [ 1279.682469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_no_route -r 1 [ 1279.773713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_no_route -r 1 [ 1279.867611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1279.906173] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1279.915891] LNet: Skipped 2 previous similar messages [ 1280.208131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1280.282748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1280.348950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1280.407674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1280.440632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1280.494555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1280.533714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1280.590743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1280.625718] LNet: 57953:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1280.650848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1280.700911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1280.740800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.776281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1280.844669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1280.904448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.982271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1281.797992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_timeout -r 1 [ 1281.900147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_timeout -r 1 [ 1281.948594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_timeout -r 1 [ 1281.987813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_timeout -r 1 [ 1282.055855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_timeout -r 1 [ 1282.157710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_timeout -r 1 [ 1282.257156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_timeout -r 1 [ 1282.306756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_timeout -r 1 [ 1282.381835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1282.844538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1282.919666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1282.951387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1282.996245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1283.076777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1283.133565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1283.209373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1283.284442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1283.337642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1283.406983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1283.452139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.537657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1283.602833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1283.655362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1283.708708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1284.429365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_error -r 1 [ 1284.472933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_error -r 1 [ 1284.534122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_error -r 1 [ 1284.624928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_error -r 1 [ 1284.695135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_error -r 1 [ 1284.745969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_error -r 1 [ 1284.777816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_error -r 1 [ 1284.816892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_error -r 1 [ 1284.856034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1284.875628] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1284.884396] LNet: Skipped 7 previous similar messages [ 1285.179952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1285.233432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1285.307590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1285.357121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1285.418237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1285.465903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1285.491879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1285.514988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1285.528995] LNet: 58196:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1285.534325] LNet: 58196:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1285.563823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1285.602909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1285.635586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1285.684515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1285.747880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.813698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1285.874585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1292.979905] LNet: 58271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1292.988953] LNetError: 58271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.047937] LNet: Removed LNI 192.168.204.10@tcp [ 1294.050377] LNetError: 58271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.074819] LNet: Removed LNI 192.168.204.10@tcp1 [ 1294.742881] Key type .llcrypt unregistered [ 1294.747778] Key type ._llcrypt unregistered [ 1316.465231] Key type ._llcrypt registered [ 1316.467249] Key type .llcrypt registered [ 1317.783237] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:55:12 (1770684912) [ 1319.013538] Key type .llcrypt unregistered [ 1319.019521] Key type ._llcrypt unregistered [ 1326.863609] Key type ._llcrypt registered [ 1326.866350] Key type .llcrypt registered [ 1327.114483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1327.148597] alg: No test for adler32 (adler32-zlib) [ 1328.158256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1328.215724] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1338.050352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1338.829217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_dropped -r 1 [ 1338.884597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_dropped -r 1 [ 1338.946993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1338.970636] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1339.328462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1339.389137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1339.447541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1339.487937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1339.521386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1339.550831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1339.577857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1339.615124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1339.629799] LNet: 59515:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1339.647787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1339.673423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1339.696981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1339.731596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1339.755590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1339.781844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1339.808672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1340.404229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_error -r 1 [ 1340.430906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_error -r 1 [ 1340.468993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1340.502430] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1340.801725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1340.847762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1340.891944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1340.925262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1340.954621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1340.998825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.041402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.071820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.095468] LNet: 59624:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.113730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.144605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.171286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.200583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.231607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.255564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.281254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1341.859889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_timeout -r 1 [ 1341.901536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_timeout -r 1 [ 1341.947582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1341.974781] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1342.322893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.345408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1342.367872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.397226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.420356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.443990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.475095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.500783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.516550] LNet: 59733:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.530767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.568982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.601714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.626379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.654283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.682620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1342.712247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1342.931890] LNet: 59773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.939090] LNetError: 59773:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.973842] LNet: Removed LNI 192.168.204.10@tcp [ 1344.398397] Key type .llcrypt unregistered [ 1344.401068] Key type ._llcrypt unregistered [ 1365.167673] Key type ._llcrypt registered [ 1365.176871] Key type .llcrypt registered [ 1366.782279] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:56:01 (1770684961) [ 1367.659431] Key type .llcrypt unregistered [ 1367.667773] Key type ._llcrypt unregistered [ 1375.250496] Key type ._llcrypt registered [ 1375.253916] Key type .llcrypt registered [ 1375.503795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1375.519996] alg: No test for adler32 (adler32-zlib) [ 1376.442995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1376.459993] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1385.233028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1395.454925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.529919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1395.555986] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1396.985224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_dropped -r 1 [ 1397.024735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e remote_dropped -r 1 [ 1397.061846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_dropped -r 1 [ 1397.097457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e remote_dropped -r 1 [ 1397.128125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e remote_dropped -r 1 [ 1397.167923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e remote_dropped -r 1 [ 1397.227978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e remote_dropped -r 1 [ 1397.274958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e remote_dropped -r 1 [ 1397.315829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1397.332395] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1397.335539] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.110@tcp [ 1397.548275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1397.577489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1397.621103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1397.663188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1397.710128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1397.742039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1397.775748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1397.816821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1397.834690] LNet: 61236:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1397.856575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1397.893444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1397.933429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1397.966635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1397.995774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1398.047544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1398.083513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1398.117843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1398.575087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_error -r 1 [ 1398.611755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e remote_error -r 1 [ 1398.661752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_error -r 1 [ 1398.707633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e remote_error -r 1 [ 1398.743855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e remote_error -r 1 [ 1398.791985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e remote_error -r 1 [ 1398.833441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e remote_error -r 1 [ 1398.870342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e remote_error -r 1 [ 1398.903696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1398.925819] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1398.933956] LNet: Skipped 2 previous similar messages [ 1399.157619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1399.190442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1399.217353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1399.242954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1399.267743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1399.300661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1399.340363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1399.365701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1399.380901] LNet: 61359:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1399.395375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1399.424763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1399.454480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1399.497897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1399.530646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1399.558300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1399.585549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1399.612781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1400.173802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_timeout -r 1 [ 1400.237891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e remote_timeout -r 1 [ 1400.309652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_timeout -r 1 [ 1400.379855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e remote_timeout -r 1 [ 1400.445518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e remote_timeout -r 1 [ 1400.516356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e remote_timeout -r 1 [ 1400.585386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e remote_timeout -r 1 [ 1400.656132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e remote_timeout -r 1 [ 1400.733792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.110@tcp [ 1400.788543] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1400.800910] LNet: Skipped 1 previous similar message [ 1401.133437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1401.157544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1401.181797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1401.206536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1401.229969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1401.254652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1401.275462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1401.300284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1401.312572] LNet: 61482:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1401.326448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1401.354645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1401.376851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1401.398889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1401.424903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1401.451649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1401.477491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1401.511416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1406.673925] LNet: 61559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.683787] LNetError: 61559:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.688868] LNet: Removed LNI 192.168.204.10@tcp [ 1406.701111] LNet: Removed LNI 192.168.204.10@tcp1 [ 1407.294823] Key type .llcrypt unregistered [ 1407.300598] Key type ._llcrypt unregistered [ 1426.959240] Key type ._llcrypt registered [ 1426.972695] Key type .llcrypt registered [ 1428.640398] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:57:03 (1770685023) [ 1429.438622] Key type .llcrypt unregistered [ 1429.447131] Key type ._llcrypt unregistered [ 1436.656369] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1436.851748] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1436.861887] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1436.866628] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1439.639646] Key type ._llcrypt registered [ 1439.641584] Key type .llcrypt registered [ 1439.838448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1439.856716] alg: No test for adler32 (adler32-zlib) [ 1440.914561] LNet: 62631:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1440.925757] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1440.955269] LNet: 62645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1440.960626] LNetError: 62645:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1440.975940] LNet: Removed LNI 192.168.204.10@tcp [ 1441.787606] Key type .llcrypt unregistered [ 1441.791255] Key type ._llcrypt unregistered [ 1448.069737] Key type ._llcrypt registered [ 1448.076754] Key type .llcrypt registered [ 1448.760111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1448.785630] alg: No test for adler32 (adler32-zlib) [ 1449.931735] LNet: 62900:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1449.986978] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1449.990362] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1450.079875] LNet: 62914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1450.097105] LNetError: 62914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1450.112655] LNet: Removed LNI 192.168.204.10@tcp [ 1450.134084] LNet: Removed LNI 10.1.2.3@tcp1 [ 1451.333968] Key type .llcrypt unregistered [ 1451.336813] Key type ._llcrypt unregistered [ 1457.475671] Key type ._llcrypt registered [ 1457.480189] Key type .llcrypt registered [ 1457.899983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1457.964217] alg: No test for adler32 (adler32-zlib) [ 1459.194842] LNet: 63169:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1459.246946] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1459.253099] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1459.346877] LNet: 63183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1459.377477] LNetError: 63183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.393961] LNet: Removed LNI 192.168.204.10@tcp [ 1459.411591] LNet: Removed LNI 10.1.2.3@tcp [ 1460.725406] Key type .llcrypt unregistered [ 1460.731476] Key type ._llcrypt unregistered [ 1468.442058] Key type ._llcrypt registered [ 1468.447670] Key type .llcrypt registered [ 1468.773270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1468.801406] alg: No test for adler32 (adler32-zlib) [ 1469.802909] LNet: 63440:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1469.847463] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1469.862652] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1469.955805] LNet: 63454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.971974] LNetError: 63454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.979128] LNet: Removed LNI 192.168.204.10@tcp [ 1469.994322] LNet: Removed LNI 10.1.2.3@tcp [ 1470.797626] Key type .llcrypt unregistered [ 1470.802898] Key type ._llcrypt unregistered [ 1479.618237] Key type ._llcrypt registered [ 1479.620646] Key type .llcrypt registered [ 1479.871984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1479.891127] alg: No test for adler32 (adler32-zlib) [ 1480.983451] LNet: 63715:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1481.008450] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1481.011505] LNetError: 63715:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1481.036441] LNetError: Error -100 starting up LNI tcp [ 1481.057848] LNet: Removed LNI 192.168.204.10@tcp [ 1481.852617] Key type .llcrypt unregistered [ 1481.858064] Key type ._llcrypt unregistered [ 1497.116265] Key type ._llcrypt registered [ 1497.118524] Key type .llcrypt registered [ 1498.618780] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:58:13 (1770685093) [ 1499.724439] Key type .llcrypt unregistered [ 1499.730290] Key type ._llcrypt unregistered [ 1507.167218] Key type ._llcrypt registered [ 1507.168987] Key type .llcrypt registered [ 1507.624413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1507.665896] alg: No test for adler32 (adler32-zlib) [ 1508.602283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1508.634087] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1517.150133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1518.284597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e network_timeout -r 1 [ 1518.360820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e network_timeout -r 1 [ 1518.453422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1518.499310] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1518.862971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1518.891112] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1518.912455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1518.955749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1518.991727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1519.044187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1519.099176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1519.152826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1519.210608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1519.242453] LNet: 64967:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1519.271272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1519.316933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1519.383271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.458260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1519.518463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1519.586881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1519.972958] LNet: 65005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.986823] LNetError: 65005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.007391] LNet: Removed LNI 192.168.204.10@tcp [ 1520.573833] Key type .llcrypt unregistered [ 1520.576287] Key type ._llcrypt unregistered [ 1542.362626] Key type ._llcrypt registered [ 1542.364232] Key type .llcrypt registered [ 1542.756084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.811523] alg: No test for adler32 (adler32-zlib) [ 1543.826421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1543.866934] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1553.722116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1563.980605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.046359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1564.087516] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1566.086833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e network_timeout -r 1 [ 1566.152302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e network_timeout -r 1 [ 1566.226160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e network_timeout -r 1 [ 1566.289811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e network_timeout -r 1 [ 1566.330729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e network_timeout -r 1 [ 1566.392366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e network_timeout -r 1 [ 1566.487352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e network_timeout -r 1 [ 1566.562780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e network_timeout -r 1 [ 1566.639752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1566.686718] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1567.061672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1567.091313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1567.121514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1567.182935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1567.228338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1567.258388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1567.302248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1567.347759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1567.368820] LNet: 65885:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1567.387285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1567.419601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1567.463523] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1567.466153] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.110@tcp [ 1567.478280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1567.480056] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1567.497139] LNet: Skipped 1 previous similar message [ 1567.540318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1567.579528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1567.631995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1572.891041] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1573.329613] LNet: 65958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.339590] LNetError: 65958:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.347042] LNet: Removed LNI 192.168.204.10@tcp [ 1573.360606] LNet: Removed LNI 192.168.204.10@tcp1 [ 1573.970658] Key type .llcrypt unregistered [ 1573.974162] Key type ._llcrypt unregistered [ 1596.055692] Key type ._llcrypt registered [ 1596.058232] Key type .llcrypt registered [ 1597.571791] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:59:52 (1770685192) [ 1597.981360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.992398] alg: No test for adler32 (adler32-zlib) [ 1599.030723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.122033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.180970] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1600.168701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1600.198082] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1601.204465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 1601.344811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1601.439492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1601.522982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1601.567340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1601.635076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1601.701372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.10@tcp [ 1601.734793] LNetError: 67058:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.10@tcp to recovery [ 1602.143509] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1602.143942] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1604.210161] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1604.402546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1608.289932] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1609.151311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1609.172695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1609.231249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.248429] LNet: 67165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.254256] LNetError: 67165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.260395] LNet: Removed LNI 192.168.204.10@tcp [ 1609.271575] LNet: Removed LNI 192.168.204.10@tcp1 [ 1609.290237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.334297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.359809] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1610.104782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1610.125697] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1610.843076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 1610.914112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1610.983776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1611.033681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1611.070299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1611.107246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1611.148649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1611.179606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.10@tcp [ 1611.214175] LNetError: 67438:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.10@tcp to recovery [ 1611.359359] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1613.676841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1617.503509] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1617.513320] LNet: Skipped 1 previous similar message [ 1621.843826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1621.893432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1621.958493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1630.005940] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:00:24 (1770685224) [ 1630.242524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.289506] LNet: 68147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1630.298451] LNetError: 68147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1630.310545] LNetError: 68147:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1630.318440] LNet: Removed LNI 192.168.204.10@tcp [ 1630.336857] LNet: Removed LNI 192.168.204.10@tcp1 [ 1630.374723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.453382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1630.479069] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1631.432560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1631.458413] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1632.482032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 1632.575801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1632.618692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1632.683504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1632.729813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1632.789441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1632.834406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.10@tcp --health 0 [ 1632.959078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1633.504064] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.10@tcp [ 1633.505412] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1633.524170] LNet: Skipped 1 previous similar message [ 1648.666265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1648.793841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1648.821780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1648.854421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.10@tcp --health 500 [ 1648.985257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1655.963907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1656.059668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.083315] LNet: 68639:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.094347] LNetError: 68639:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.098629] LNetError: 68639:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1656.107981] LNet: Removed LNI 192.168.204.10@tcp [ 1656.131225] LNet: Removed LNI 192.168.204.10@tcp1 [ 1656.165667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.234380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1656.266266] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1657.123947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1657.140651] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1658.265986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 1658.362097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1658.491644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1658.547262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1658.614244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1658.672888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1658.732666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1658.765306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.10@tcp --health 0 [ 1659.296729] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1659.306235] LNet: Skipped 3 previous similar messages [ 1660.225671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1670.679824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1670.709554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1670.736766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1679.007637] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:01:13 (1770685273) [ 1679.431590] LNet: 69699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.435819] LNetError: 69699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.438844] LNetError: 69699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1679.444492] LNet: Removed LNI 192.168.204.10@tcp [ 1679.475995] LNet: Removed LNI 192.168.204.10@tcp1 [ 1680.247215] Key type .llcrypt unregistered [ 1680.249736] Key type ._llcrypt unregistered [ 1687.712897] Key type ._llcrypt registered [ 1687.715452] Key type .llcrypt registered [ 1687.995417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1688.022720] alg: No test for adler32 (adler32-zlib) [ 1689.013656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1689.057794] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1700.193529] Lustre: DEBUG MARKER: Initial discovery [ 1700.290231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.110@tcp [ 1703.211739] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1703.272698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1703.325224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.110@tcp [ 1703.366872] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1703.398432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1703.452970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.110@tcp -r 1 -m GET -l 3 [ 1703.464220] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1703.492367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.110@tcp -r 1 -m GET -e local_error [ 1703.532769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.110@tcp -r 1 -m PUT -l 6 [ 1705.238398] Lustre: DEBUG MARKER: Force oleg410-client.virtnet to discover 192.168.204.110@tcp (in background) [ 1705.535408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.110@tcp [ 1707.047458] Lustre: DEBUG MARKER: Force oleg410-server to discover 192.168.204.10@tcp [ 1708.639338] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1710.263979] Lustre: DEBUG MARKER: Wait for 70459 [ 1711.691203] Lustre: DEBUG MARKER: Finished wait on 70459 [ 1712.015925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1712.078622] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1712.084814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1712.448607] LNet: 70908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1712.466159] LNetError: 70908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.518314] LNet: Removed LNI 192.168.204.10@tcp [ 1714.136578] Key type .llcrypt unregistered [ 1714.138800] Key type ._llcrypt unregistered [ 1735.464982] Key type ._llcrypt registered [ 1735.467946] Key type .llcrypt registered [ 1737.081844] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:02:11 (1770685331) [ 1737.827414] Key type .llcrypt unregistered [ 1737.834177] Key type ._llcrypt unregistered [ 1745.594342] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1745.838090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1745.848738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1745.863417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1748.587983] Key type ._llcrypt registered [ 1748.590949] Key type .llcrypt registered [ 1748.879372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1748.915821] alg: No test for adler32 (adler32-zlib) [ 1749.927416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1750.056878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1750.080899] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1751.039657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1751.070341] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1760.935383] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:02:35 (1770685355) [ 1761.224865] LNet: 72995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1761.231483] LNetError: 72995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.238477] LNet: Removed LNI 192.168.204.10@tcp [ 1761.267072] LNet: Removed LNI 10.1.2.3@tcp [ 1762.128630] Key type .llcrypt unregistered [ 1762.131052] Key type ._llcrypt unregistered [ 1770.910246] Key type ._llcrypt registered [ 1770.923293] Key type .llcrypt registered [ 1771.192770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1771.215740] alg: No test for adler32 (adler32-zlib) [ 1772.233487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.337868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.358045] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1773.358366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1773.394863] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1774.797148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.10@tcp [ 1775.903192] LNetError: Unexpected error -22 connecting to 192.168.204.10@tcp at host 192.168.204.10:988 [ 1782.775423] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:02:57 (1770685377) [ 1782.917048] LNet: 74116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.931503] LNetError: 74116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.946330] LNet: Removed LNI 192.168.204.10@tcp [ 1783.003128] LNet: Removed LNI 10.1.2.3@tcp [ 1783.731090] Key type .llcrypt unregistered [ 1783.732709] Key type ._llcrypt unregistered [ 1791.320883] Key type ._llcrypt registered [ 1791.325902] Key type .llcrypt registered [ 1791.579838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1791.607568] alg: No test for adler32 (adler32-zlib) [ 1792.527318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.572992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1792.600402] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1793.401865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1793.415974] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 1794.214821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.10@tcp1 --nid 192.168.204.10@tcp2 [ 1798.964706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp1 192.168.204.10@tcp2 [ 1799.010321] LNetError: 74956:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.10@tcp1 to 192.168.204.10@tcp2 - Route table may be misconfigured [ 1799.049838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp2 192.168.204.10@tcp1 [ 1806.644899] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:03:21 (1770685401) [ 1806.728504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.747376] LNet: 75522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.762433] LNetError: 75522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.839207] LNet: Removed LNI 192.168.204.10@tcp1 [ 1807.848194] LNetError: 75522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.882320] LNet: Removed LNI 192.168.204.10@tcp2 [ 1807.939296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.027290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.085286] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1808.915396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1808.933124] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1809.685837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 1809.713048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.10@tcp [ 1809.745202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp1 192.168.204.10@tcp1 [ 1809.777334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.10@tcp -d 192.168.204.10@tcp -e network_timeout [ 1809.809861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -e network_timeout [ 1809.844492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -e network_timeout [ 1809.881513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -e network_timeout [ 1809.913800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1809.933598] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 1809.950215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1810.040460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1810.084418] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1810.146290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1819.842474] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:03:33 (1770685413) [ 1820.253727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.272420] LNet: 76392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.279472] LNetError: 76392:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.286932] LNet: Removed LNI 192.168.204.10@tcp [ 1821.351458] LNet: Removed LNI 192.168.204.10@tcp1 [ 1821.397956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.479234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1821.836538] LNet: 76428:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.415708] Key type .llcrypt unregistered [ 1822.416875] Key type ._llcrypt unregistered [ 1837.123191] Key type ._llcrypt registered [ 1837.127108] Key type .llcrypt registered [ 1838.925476] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:03:53 (1770685433) [ 1839.596225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1839.632944] alg: No test for adler32 (adler32-zlib) [ 1840.540878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.316791] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1844.423975] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:03:58 (1770685438) [ 1844.594509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.638099] LNet: 77518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.689173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.810758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.850977] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1846.103525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1846.139227] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1847.215838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1847.258344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp1 [ 1847.295775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp1 [ 1855.303764] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:04:09 (1770685449) [ 1857.525104] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg410-server [ 1859.192091] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:04:14 (1770685454) [ 1861.557505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg410-server [ 1863.743151] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:04:18 (1770685458) [ 1865.956434] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg410-server [ 1868.031605] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:04:22 (1770685462) [ 1869.916166] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg410-server [ 1872.093415] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:04:26 (1770685466) [ 1874.156984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg410-server [ 1876.223865] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:04:30 (1770685470) [ 1878.430700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg410-server [ 1880.324660] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:04:35 (1770685475) [ 1882.688244] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg410-server [ 1884.882751] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:04:39 (1770685479) [ 1887.048261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg410-server [ 1888.825229] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:04:43 (1770685483) [ 1891.341129] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg410-server [ 1892.811889] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:04:47 (1770685487) [ 1892.954373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.994979] LNet: 81404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.006585] LNetError: 81404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.016385] LNet: Removed LNI 192.168.204.10@tcp [ 1894.064913] LNet: Removed LNI 192.168.204.10@tcp1 [ 1894.123309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.187383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1894.220072] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1894.937645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1895.004601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1895.135512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1895.157575] LNet: 81552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1895.162948] LNetError: 81552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.168323] LNetError: 81552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1895.184949] LNet: Removed LNI 192.168.204.10@tcp [ 1895.211112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1895.278739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1895.328957] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1896.135769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1896.336138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.362096] LNetError: 81697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.387328] LNet: Removed LNI 192.168.204.10@tcp [ 1896.438987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.542398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.567471] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1897.377669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1897.484327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1900.006436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.035677] LNet: 81863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.046786] LNet: 81863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1900.056459] LNetError: 81863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1901.108135] LNet: Removed LNI 192.168.204.10@tcp [ 1901.156815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.257867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1901.277687] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1902.150840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1902.415555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.476390] LNet: Removed LNI 192.168.204.10@tcp [ 1903.529673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.616636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1903.652844] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1904.548821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1904.616105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1904.876016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.909734] LNet: 82156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.915937] LNet: 82156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1904.922981] LNetError: 82156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.927149] LNetError: 82156:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1904.962116] LNet: Removed LNI 192.168.204.10@tcp [ 1905.027556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.103279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1905.165159] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1906.175411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1906.408630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.495800] LNet: Removed LNI 192.168.204.10@tcp [ 1907.542691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.621141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1907.654909] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1908.424355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1908.529261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1908.801847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.862144] LNet: Removed LNI 192.168.204.10@tcp [ 1909.897967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.973902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1910.016870] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1910.897209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1911.261878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.358700] LNet: Removed LNI 192.168.204.10@tcp [ 1912.439901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.573605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1912.637596] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1913.570701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1913.689442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1914.009729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.062902] LNet: 82742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.065926] LNet: 82742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1914.076665] LNetError: 82742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.081981] LNetError: 82742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1914.101826] LNet: Removed LNI 192.168.204.10@tcp [ 1914.144993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.239470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1914.273023] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1915.126505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1915.487844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.585949] LNet: Removed LNI 192.168.204.10@tcp [ 1916.686286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.752609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.804458] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1917.411389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1917.504409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1917.749964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.828556] LNet: Removed LNI 192.168.204.10@tcp [ 1918.857227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.936509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1918.972606] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1920.003855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 1920.400418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.446426] LNet: Removed LNI 192.168.204.10@tcp [ 1921.473240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.545218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.581356] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1922.243756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1922.300357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.347620] LNet: Removed LNI 192.168.204.10@tcp [ 1922.369485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.416406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.439443] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1923.078841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1930.548447] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:05:25 (1770685525) [ 1930.691479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.720521] LNet: 84015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.729327] LNet: 84015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1930.738720] LNetError: 84015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.747148] LNetError: 84015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1930.775799] LNet: Removed LNI 192.168.204.10@tcp [ 1930.830272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.873404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1930.892044] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/180] [ 1932.426044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.456507] LNet: Removed LNI 192.168.204.10@tcp231 [ 1932.484779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.652176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1932.709489] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/180] [ 1932.731869] LNet: 84164:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1933.352648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1933.390975] LNet: Removed LNI 192.168.204.10@tcp231 [ 1933.431562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1933.466289] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/180] [ 1934.603538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.639547] LNet: Removed LNI 192.168.204.10@tcp231 [ 1934.664204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.770226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1934.806061] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/0] [ 1934.809565] LNet: 84316:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1935.148520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1935.176155] LNet: Removed LNI 192.168.204.10@tcp231 [ 1935.192873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1935.223856] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/0] [ 1936.566360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.623983] LNet: Removed LNI 192.168.204.10@tcp231 [ 1936.641707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1936.684110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1936.700556] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/60] [ 1936.704755] LNet: 84467:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1937.074867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1937.103271] LNet: Removed LNI 192.168.204.10@tcp231 [ 1937.121608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1937.168652] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/60] [ 1938.553557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.602873] LNet: Removed LNI 192.168.204.10@tcp231 [ 1938.626949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.736257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1938.782419] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/180] [ 1938.789722] LNet: 84619:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1939.325944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1939.359321] LNet: Removed LNI 192.168.204.10@tcp231 [ 1939.374489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1939.403905] LNet: Added LNI 192.168.204.10@tcp231 [8/256/0/180] [ 1947.373822] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:05:42 (1770685542) [ 1947.686986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.728158] LNet: Removed LNI 192.168.204.10@tcp231 [ 1947.769894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.826129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1947.856443] LNet: Added LNI 192.168.204.10@tcp232 [8/256/0/180] [ 1948.581601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1948.615992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1948.644163] LNet: Removed LNI 192.168.204.10@tcp232 [ 1948.678685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1948.720074] LNet: Added LNI 192.168.204.10@tcp232 [8/256/0/180] [ 1949.554873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1949.599824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1949.642333] LNet: Removed LNI 192.168.204.10@tcp232 [ 1949.663709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1949.716757] LNet: Added LNI 192.168.204.10@tcp232 [8/256/0/180] [ 1950.576831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1950.608111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1950.647734] LNet: Removed LNI 192.168.204.10@tcp232 [ 1950.671758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1950.716520] LNet: Added LNI 192.168.204.10@tcp232 [8/256/0/180] [ 1951.911814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1952.006739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1952.054959] LNet: Removed LNI 192.168.204.10@tcp232 [ 1952.090761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1958.684695] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:05:53 (1770685553) [ 1959.728823] Key type .llcrypt unregistered [ 1959.732624] Key type ._llcrypt unregistered [ 1966.696820] Key type ._llcrypt registered [ 1966.705372] Key type .llcrypt registered [ 1967.010584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1967.033588] alg: No test for adler32 (adler32-zlib) [ 1968.057124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1968.110876] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 1975.296488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 1984.162372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.234976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1984.267495] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 1985.768044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_interrupt -r 2 [ 1985.821420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_interrupt -r 2 [ 1985.879920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_interrupt -r 2 [ 1985.931673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_interrupt -r 2 [ 1985.981972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_interrupt -r 2 [ 1986.046885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_interrupt -r 2 [ 1986.096714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_interrupt -r 2 [ 1986.140390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_interrupt -r 2 [ 1986.212958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.110@tcp [ 1986.233503] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1986.238032] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 1986.455078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1986.484171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1986.506400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1986.537562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1986.565099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1986.588417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1986.614154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1986.643174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1986.661426] LNet: 87119:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1986.679799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1986.713772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1986.753508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1986.785388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1986.813857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1986.866066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1987.436227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_dropped -r 2 [ 1987.466985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_dropped -r 2 [ 1987.494988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_dropped -r 2 [ 1987.521980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_dropped -r 2 [ 1987.565104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_dropped -r 2 [ 1987.612488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_dropped -r 2 [ 1987.697531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_dropped -r 2 [ 1987.783191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_dropped -r 2 [ 1987.828272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.110@tcp [ 1987.851580] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1988.143388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1988.176644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1988.202386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1988.226987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1988.251384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1988.276883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1988.299778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1988.346595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1988.363778] LNet: 87239:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1988.380839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1988.407128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1988.458840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1988.507406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1988.553389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1988.609971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1989.176496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_aborted -r 2 [ 1989.210637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_aborted -r 2 [ 1989.237519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_aborted -r 2 [ 1989.273799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_aborted -r 2 [ 1989.310626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_aborted -r 2 [ 1989.354875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_aborted -r 2 [ 1989.395537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_aborted -r 2 [ 1989.423820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_aborted -r 2 [ 1989.474933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.110@tcp [ 1989.493250] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1989.792297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1989.856767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1989.901284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1989.940728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1989.989206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1990.059132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1990.108367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1990.161084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1990.173238] LNet: 87360:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1990.189894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1990.223530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1990.280171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1990.329360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1990.364270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1990.393645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1991.040567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_no_route -r 2 [ 1991.085643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_no_route -r 2 [ 1991.127670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_no_route -r 2 [ 1991.170145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_no_route -r 2 [ 1991.198570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_no_route -r 2 [ 1991.235809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_no_route -r 2 [ 1991.273837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_no_route -r 2 [ 1991.310591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_no_route -r 2 [ 1991.343338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.110@tcp [ 1991.664600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1991.688860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1991.723723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1991.767513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1991.851242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1991.904749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1991.936839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1991.965780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1992.019861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1992.068393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1992.121632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1992.173940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1992.240243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1992.286614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1992.816711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e local_timeout -r 2 [ 1992.853298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e local_timeout -r 2 [ 1992.883875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e local_timeout -r 2 [ 1992.921993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e local_timeout -r 2 [ 1992.960949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e local_timeout -r 2 [ 1993.007497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e local_timeout -r 2 [ 1993.061474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e local_timeout -r 2 [ 1993.093586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e local_timeout -r 2 [ 1993.133322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.110@tcp [ 1993.151197] LNet: There was an unexpected network error while writing to 192.168.204.110: rc = -22 [ 1993.159903] LNet: Skipped 1 previous similar message [ 1993.481985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1993.516754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1993.552919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1993.595981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1993.632051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1993.697235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1993.747321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1993.797332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1993.856084] LNet: 87600:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1993.864527] LNet: 87600:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1993.890991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1993.964588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1993.994332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1994.029277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1994.057695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1994.111749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1994.832254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp -m GET -e remote_dropped -r 2 [ 1994.860415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.110@tcp1 -m GET -e remote_dropped -r 2 [ 1994.881965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp -m GET -e remote_dropped -r 2 [ 1994.921814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp -d 192.168.204.10@tcp1 -m GET -e remote_dropped -r 2 [ 1994.951968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp -m GET -e remote_dropped -r 2 [ 1994.983571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.110@tcp1 -m GET -e remote_dropped -r 2 [ 1995.018959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp -m GET -e remote_dropped -r 2 [ 1995.054957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.10@tcp1 -d 192.168.204.10@tcp1 -m GET -e remote_dropped -r 2 [ 1995.086814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.10@tcp 192.168.204.110@tcp [ 1995.118388] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.110@tcp [ 1995.322742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1995.376227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1995.437287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1995.480114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1995.519211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1995.558834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1995.608847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1995.646675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1995.694821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1995.752993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1995.807869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1995.836265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1995.874394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1995.917121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2001.789917] LNet: 87794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2001.805234] LNetError: 87794:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.813619] LNet: Removed LNI 192.168.204.10@tcp [ 2001.832152] LNet: Removed LNI 192.168.204.10@tcp1 [ 2002.233989] Key type .llcrypt unregistered [ 2002.237421] Key type ._llcrypt unregistered [ 2020.093734] Key type ._llcrypt registered [ 2020.101298] Key type .llcrypt registered [ 2021.166958] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:06:56 (1770685616) [ 2021.526780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.551659] alg: No test for adler32 (adler32-zlib) [ 2022.467916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2022.514862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2022.554589] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2023.307194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2023.328899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.10@tcp [ 2023.343520] LNetError: 88770:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.10@tcp to recovery [ 2023.415470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.10@tcp [ 2023.501582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2029.764927] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:07:04 (1770685624) [ 2029.965296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2030.030370] LNet: 89359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2030.045473] LNetError: 89359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2030.073897] LNet: Removed LNI 192.168.204.10@tcp [ 2030.097815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.132436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2030.164154] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2030.762457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2030.783334] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 2031.645767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.10@tcp [ 2031.663433] LNetError: 89625:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.10@tcp to recovery [ 2032.159377] LNet: 1 local NIs in recovery (showing 1): 192.168.204.10@tcp [ 2032.161290] LNet: There was an unexpected network error while writing to 192.168.204.10: rc = -22 [ 2032.891780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.10@tcp [ 2041.080273] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:07:15 (1770685635) [ 2042.914430] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg410-server [ 2045.126388] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:07:19 (1770685639) [ 2045.236216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2045.278562] LNet: 90576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.288111] LNetError: 90576:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.295801] LNet: Removed LNI 192.168.204.10@tcp [ 2045.322773] LNet: Removed LNI 192.168.204.10@tcp1 [ 2045.374399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2045.420921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2045.447004] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2046.322765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2046.409635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2046.498407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2046.588191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2046.627969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2046.666362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2053.119225] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:07:28 (1770685648) [ 2053.484686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2053.530177] LNet: 91307:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2053.538914] LNetError: 91307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.552181] LNetError: 91307:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2053.576996] LNet: Removed LNI 192.168.204.10@tcp [ 2053.614557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.695167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.714870] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2060.671537] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:07:35 (1770685655) [ 2060.765874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.789059] LNet: 92005:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.801112] LNetError: 92005:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.828350] LNet: Removed LNI 192.168.204.10@tcp [ 2060.865605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.919573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2060.947490] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2067.685324] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:07:42 (1770685662) [ 2069.273729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2070.831961] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:07:45 (1770685665) [ 2071.266913] LNet: 93045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2071.271566] LNetError: 93045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2071.285950] LNet: Removed LNI 192.168.204.10@tcp [ 2071.846857] Key type .llcrypt unregistered [ 2071.848590] Key type ._llcrypt unregistered [ 2078.527909] Key type ._llcrypt registered [ 2078.530848] Key type .llcrypt registered [ 2078.716648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2078.734957] alg: No test for adler32 (adler32-zlib) [ 2079.726406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2079.749874] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2087.649242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 2092.340724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.110@tcp [ 2092.699811] LNet: 93458:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.713884] LNetError: 93458:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.739817] LNet: Removed LNI 192.168.204.10@tcp [ 2093.082834] Key type .llcrypt unregistered [ 2093.086232] Key type ._llcrypt unregistered [ 2105.206574] Key type ._llcrypt registered [ 2105.209345] Key type .llcrypt registered [ 2106.387643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2107.834731] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2109.747959] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:08:24 (1770685704) [ 2110.213413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2110.220758] alg: No test for adler32 (adler32-zlib) [ 2111.172030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.269813] LNet: 94542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.914441] Key type .llcrypt unregistered [ 2111.918058] Key type ._llcrypt unregistered [ 2117.753902] Key type ._llcrypt registered [ 2117.756946] Key type .llcrypt registered [ 2118.016574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2118.045486] alg: No test for adler32 (adler32-zlib) [ 2119.015551] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2119.018129] LNet: 94794:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 2119.024665] LNet: 94794:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2119.055944] LNetError: 94806:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.12@tcp is being used as a gateway but routing feature is not turned on [ 2119.077171] LNetError: 94806:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.204.12@tcp has gone from up to down [ 2119.472231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2119.496383] LNet: 94852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2119.502507] LNetError: 94852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2119.524371] LNet: Removed LNI 192.168.204.10@tcp [ 2119.544807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.607962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2119.631326] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2119.635992] LNet: 94859:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 2119.640236] LNet: 94859:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 2119.643916] LNetError: 94859:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.12@tcp is a gateway. Can not delete it [ 2119.649577] LNetError: 94859:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.204.12@tcp is a router. Can not be deleted [ 2119.659580] LNet: 94859:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2119.688520] LNetError: 94856:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.13@tcp is being used as a gateway but routing feature is not turned on [ 2119.703812] LNetError: 94856:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 2119.717677] LNetError: 94856:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.204.13@tcp has gone from up to down [ 2119.730750] LNetError: 94856:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 2125.552137] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:08:40 (1770685720) [ 2126.925876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg410-server [ 2128.481645] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:08:43 (1770685723) [ 2130.165561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg410-server [ 2131.492925] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:08:46 (1770685726) [ 2133.030028] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2134.855290] Key type lgssc registered [ 2136.014684] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2154.284785] Key type lgssc unregistered [ 2154.731989] LNet: 97278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.746355] LNetError: 97278:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.775191] LNet: Removed LNI 192.168.204.10@tcp [ 2155.360800] Key type .llcrypt unregistered [ 2155.363710] Key type ._llcrypt unregistered [ 2171.229697] Key type ._llcrypt registered [ 2171.231032] Key type .llcrypt registered [ 2172.554136] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:09:27 (1770685767) [ 2172.889393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.901445] alg: No test for adler32 (adler32-zlib) [ 2173.936123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2174.046893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2174.100111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2174.161256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2174.230366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2181.404709] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:09:36 (1770685776) [ 2181.510750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2181.540580] LNet: 98676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.591724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.686875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.734889] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2182.834673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2182.855848] LNet: 98825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.869757] LNetError: 98825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.882903] LNet: Removed LNI 192.168.204.10@tcp [ 2182.910699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.965305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2182.984520] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2183.411194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2183.428842] LNetError: 98854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2183.444617] LNet: Removed LNI 192.168.204.10@tcp [ 2183.460743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2183.507704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2183.532544] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2190.055986] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:09:44 (1770685784) [ 2190.245691] LNet: 99445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2190.251667] LNet: 99445:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2190.263758] LNetError: 99445:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2190.291164] LNet: Removed LNI 192.168.204.10@tcp [ 2191.110867] Key type .llcrypt unregistered [ 2191.114280] Key type ._llcrypt unregistered [ 2198.318514] Key type ._llcrypt registered [ 2198.320994] Key type .llcrypt registered [ 2198.501330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2198.539776] alg: No test for adler32 (adler32-zlib) [ 2199.578034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.666395] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2199.670857] LNet: *** cfs_fail_loc=e003, val=100*** [ 2200.525402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 2200.559737] LNet: *** cfs_fail_loc=e003, val=100*** [ 2200.566489] LNet: Skipped 2 previous similar messages [ 2200.596132] Ping reply received with mapping: [ 2200.598859] found metadata reply NI[192.168.204.10@tcp] [ 2200.601649] 0a cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2200.603780] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2200.608369] 5a 5a 5a 5a 5a 5a 5a 5a [ 2200.843192] LNet: 99840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2200.854869] LNetError: 99840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2200.876412] LNet: Removed LNI 192.168.204.10@tcp [ 2201.870445] Key type .llcrypt unregistered [ 2201.871774] Key type ._llcrypt unregistered [ 2214.830194] Key type ._llcrypt registered [ 2214.834191] Key type .llcrypt registered [ 2216.102743] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:10:10 (1770685810) [ 2216.752368] Key type .llcrypt unregistered [ 2216.753977] Key type ._llcrypt unregistered [ 2223.706783] Key type ._llcrypt registered [ 2223.710509] Key type .llcrypt registered [ 2223.979570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2224.010515] alg: No test for adler32 (adler32-zlib) [ 2225.226770] LNetError: 100875:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2225.620713] Key type .llcrypt unregistered [ 2225.622398] Key type ._llcrypt unregistered [ 2230.295648] Key type ._llcrypt registered [ 2230.299260] Key type .llcrypt registered [ 2231.443395] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:10:26 (1770685826) [ 2231.915566] Key type .llcrypt unregistered [ 2231.919451] Key type ._llcrypt unregistered [ 2237.798816] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2237.805297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2240.539140] Key type ._llcrypt registered [ 2240.540308] Key type .llcrypt registered [ 2240.753759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2240.767057] alg: No test for adler32 (adler32-zlib) [ 2241.849931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2241.931985] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2251.538344] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:10:46 (1770685846) [ 2251.709675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.743390] LNet: 102648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.755857] LNetError: 102648:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2251.778139] LNet: Removed LNI 10.1.2.3@tcp [ 2251.797528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.414661] Key type .llcrypt unregistered [ 2252.416672] Key type ._llcrypt unregistered [ 2259.092340] Key type ._llcrypt registered [ 2259.099696] Key type .llcrypt registered [ 2259.310442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2259.331836] alg: No test for adler32 (adler32-zlib) [ 2260.272112] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2260.377509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 2260.552853] LNet: 102932:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.559588] LNetError: 102932:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.587889] LNet: Removed LNI 192.168.204.10@tcp [ 2261.324655] Key type .llcrypt unregistered [ 2261.329385] Key type ._llcrypt unregistered [ 2273.879578] Key type ._llcrypt registered [ 2273.881714] Key type .llcrypt registered [ 2275.308329] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:11:10 (1770685870) [ 2275.953328] Key type .llcrypt unregistered [ 2275.956394] Key type ._llcrypt unregistered [ 2281.838689] Key type ._llcrypt registered [ 2281.842507] Key type .llcrypt registered [ 2282.104777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2282.118918] alg: No test for adler32 (adler32-zlib) [ 2289.124749] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:11:24 (1770685884) [ 2289.340668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2289.419507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2289.508368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2289.529487] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2290.117258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2290.147677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2290.176926] LNet: Removed LNI 192.168.204.10@tcp [ 2290.388867] LNet: 104668:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.778745] Key type .llcrypt unregistered [ 2290.785545] Key type ._llcrypt unregistered [ 2302.288141] Key type ._llcrypt registered [ 2302.290425] Key type .llcrypt registered [ 2303.685696] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:11:38 (1770685898) [ 2304.195477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2304.207410] alg: No test for adler32 (adler32-zlib) [ 2305.170913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.228624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2305.251308] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2306.215835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 2306.247740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.10@tcp [ 2306.264970] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2306.269505] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2306.272514] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2306, 0, 0 [ 2306.276579] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2306.278511] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2306.281134] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2306.283340] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2306, 0, 0 [ 2306.286612] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2306.289395] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2306.292085] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2306.294611] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2306, 0, 2356 [ 2306.297627] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2306.299404] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2306.301385] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2306.303275] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2306, 0, 2356 [ 2306.306426] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2312.232810] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:11:47 (1770685907) [ 2312.538618] LNet: 106204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.547612] LNetError: 106204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2313.576935] LNet: Removed LNI 192.168.204.10@tcp [ 2314.087444] Key type .llcrypt unregistered [ 2314.089839] Key type ._llcrypt unregistered [ 2320.615228] Key type ._llcrypt registered [ 2320.618191] Key type .llcrypt registered [ 2320.843455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2320.866263] alg: No test for adler32 (adler32-zlib) [ 2321.880976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2321.928365] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2330.114492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 2340.235092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2340.302406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2340.334077] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 2341.431866] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2341.549299] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2341.565654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2341.571218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2343.968285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2343.995897] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2356.156358] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:12:31 (1770685951) [ 2356.464086] LNet: 107709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.473745] LNetError: 107709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.485589] LNet: Removed LNI 192.168.204.10@tcp [ 2357.535294] LNet: Removed LNI 192.168.204.10@tcp1 [ 2357.545065] LNetError: 107709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2357.560441] LNetError: 107709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2357.583901] LNet: Removed LNI 10.1.2.3@tcp99 [ 2358.305337] Key type .llcrypt unregistered [ 2358.313121] Key type ._llcrypt unregistered [ 2364.964281] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2364.979531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2367.584905] Key type ._llcrypt registered [ 2367.586734] Key type .llcrypt registered [ 2367.780087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2367.803695] alg: No test for adler32 (adler32-zlib) [ 2368.724459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.801416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2368.837415] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2369.772097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2369.793144] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2372.296622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2372.340629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2372.372533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2372.389655] LNetError: 108449:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2372.417381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2372.450824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2372.825828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2372.937223] LNet: 108469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2372.945984] LNetError: 108469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2373.983869] LNet: Removed LNI 192.168.204.10@tcp [ 2373.990560] LNetError: 108469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2374.015518] LNet: Removed LNI 10.1.2.3@tcp [ 2374.902825] Key type .llcrypt unregistered [ 2374.904347] Key type ._llcrypt unregistered [ 2386.061336] Key type ._llcrypt registered [ 2386.062746] Key type .llcrypt registered [ 2387.392677] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:13:02 (1770685982) [ 2387.815564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2387.838614] alg: No test for adler32 (adler32-zlib) [ 2388.786981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2388.858992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2388.917204] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2389.717699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.10@tcp [ 2389.819599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg410-client@tcp [ 2396.558565] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:13:11 (1770685991) [ 2396.793109] LNet: 109999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.802723] LNetError: 109999:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.831645] LNet: Removed LNI 192.168.204.10@tcp [ 2397.594650] Key type .llcrypt unregistered [ 2397.597650] Key type ._llcrypt unregistered [ 2402.969867] Key type ._llcrypt registered [ 2402.972235] Key type .llcrypt registered [ 2403.112537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2403.127268] alg: No test for adler32 (adler32-zlib) [ 2404.061692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2404.100701] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2404.135297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2409.920360] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:13:25 (1770686005) [ 2411.315276] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2412.582978] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:13:27 (1770686007) [ 2413.839821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2415.355830] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:13:30 (1770686010) [ 2415.681740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2415.711483] LNet: 111415:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2415.720762] LNetError: 111415:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2415.748303] LNet: Removed LNI 192.168.204.10@tcp [ 2415.764906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2415.822255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2415.865400] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2416.449602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2416.491964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2421.680871] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:13:37 (1770686017) [ 2421.929265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2421.951301] LNet: 112124:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2421.964061] LNetError: 112124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2421.977042] LNet: Removed LNI 192.168.204.10@tcp [ 2421.990930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2422.032478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2422.054269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2422.719682] Key type .llcrypt unregistered [ 2422.724199] Key type ._llcrypt unregistered [ 2432.850920] Key type ._llcrypt registered [ 2432.853437] Key type .llcrypt registered [ 2434.031534] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:13:49 (1770686029) [ 2434.558649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2434.573401] alg: No test for adler32 (adler32-zlib) [ 2435.477387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2435.538048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2435.564314] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2436.152409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2436.218139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2441.469729] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:13:56 (1770686036) [ 2441.639449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.657397] LNet: 113676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.664106] LNetError: 113676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2441.677616] LNet: Removed LNI 192.168.204.10@tcp [ 2441.691588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2441.731113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2441.758153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2446.352881] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:14:01 (1770686041) [ 2446.455096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.493687] LNet: 114252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.540358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.602588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2446.830642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2446.856782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2446.910491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2447.167074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2447.201703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2447.266583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2447.290472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.358935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2447.701890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2447.730834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2447.773041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2448.128940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2448.193395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2454.172214] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:14:09 (1770686049) [ 2454.335228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2454.366161] LNet: 114888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.376280] LNet: 114888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2454.412187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2454.450465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2454.469489] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2455.075059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2455.094633] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2455.860196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2455.913566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2456.139956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2456.159721] LNetError: 115169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.183610] LNet: Removed LNI 192.168.204.10@tcp [ 2457.191083] LNetError: 115169:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2457.207204] LNet: Removed LNI 192.168.204.10@tcp2 [ 2457.240958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2457.298384] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2457.309931] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2457.316912] LNet: 115175:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2457.882754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2458.123424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2458.145549] LNet: 115212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2458.150666] LNet: 115212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2459.167168] LNet: Removed LNI 192.168.204.10@tcp [ 2459.176672] LNet: Removed LNI 192.168.204.10@tcp2 [ 2459.223547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2459.272560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2459.295137] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2460.087609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2460.101709] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2460.977265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2461.017496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2461.071781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2461.239655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2461.257293] LNetError: 115497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.261844] LNetError: 115497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2461.270188] LNet: Removed LNI 192.168.204.10@tcp [ 2462.318364] LNet: Removed LNI 192.168.204.10@tcp2 [ 2462.345720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2462.417422] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2462.437909] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2462.447587] LNet: 115503:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2463.079107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2463.243532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2463.258729] LNet: 115540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.261940] LNet: 115540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2463.270759] LNet: Removed LNI 192.168.204.10@tcp [ 2464.297642] LNet: Removed LNI 192.168.204.10@tcp2 [ 2464.327970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.382835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2464.400219] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2464.987767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2464.996850] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2465.829561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2465.899483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2466.115574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2466.140747] LNetError: 115820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2466.146644] LNetError: 115820:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2466.150757] LNet: Removed LNI 192.168.204.10@tcp [ 2466.158985] LNet: Removed LNI 192.168.204.10@tcp2 [ 2466.175806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.223178] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2466.230519] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2466.234202] LNet: 115826:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2466.697235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2466.885768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.935151] LNet: Removed LNI 192.168.204.10@tcp [ 2467.947968] LNet: Removed LNI 192.168.204.10@tcp2 [ 2468.005577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.113980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2468.159606] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2468.784531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2468.814796] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2469.613523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2469.646329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2469.696280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2469.863160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2470.879164] LNet: Removed LNI 192.168.204.10@tcp [ 2471.915796] LNet: Removed LNI 192.168.204.10@tcp2 [ 2471.956927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2472.077668] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2472.085569] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2472.103668] LNet: 116154:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2472.628314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2472.828417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2472.860666] LNet: 116191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.867817] LNet: 116191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2472.872785] LNet: Removed LNI 192.168.204.10@tcp [ 2473.896567] LNet: Removed LNI 192.168.204.10@tcp2 [ 2473.945864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.012182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2474.036360] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2474.720510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2474.746874] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2475.593545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2475.642149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2475.695440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2476.148631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2476.167190] LNetError: 116488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.183160] LNetError: 116488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2476.188806] LNet: Removed LNI 192.168.204.10@tcp [ 2476.195193] LNet: Removed LNI 192.168.204.10@tcp2 [ 2476.229817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2476.299854] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2476.306071] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2476.310495] LNet: 116494:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2476.808218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2477.037453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2478.111155] LNet: Removed LNI 192.168.204.10@tcp [ 2478.127845] LNet: Removed LNI 192.168.204.10@tcp2 [ 2478.153063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2478.247958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2478.296320] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2478.951935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2478.969526] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2479.735979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2479.769532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2479.800422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2479.856629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2480.190583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.247434] LNet: Removed LNI 192.168.204.10@tcp [ 2482.282951] LNet: Removed LNI 192.168.204.10@tcp2 [ 2482.344731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2482.400265] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2482.405737] LNet: Added LNI 192.168.204.10@tcp2 [8/256/0/180] [ 2482.954468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2489.097161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2490.745601] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:14:45 (1770686085) [ 2491.076939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.104423] LNet: 117613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.107216] LNet: 117613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2491.118739] LNet: Removed LNI 192.168.204.10@tcp [ 2492.135149] LNet: Removed LNI 192.168.204.10@tcp2 [ 2492.168880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.470670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2492.496110] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2493.023391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2493.046247] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2494.021919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2494.042974] LNetError: 117886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2494.051257] LNetError: 117886:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2494.062110] LNet: Removed LNI 192.168.204.10@tcp [ 2494.073380] LNet: Removed LNI 10.1.2.3@tcp [ 2494.095567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2494.218184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2494.248522] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2494.251789] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2494.782951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2494.805280] LNet: Removed LNI 192.168.204.10@tcp [ 2494.819304] LNet: Removed LNI 10.1.2.3@tcp [ 2500.763339] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:14:55 (1770686095) [ 2500.966555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2501.006591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.093302] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2501.193581] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2501.197659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2501.206550] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2503.654990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2503.683985] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2504.430964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2504.467477] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2506.118454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.10@tcp [ 2506.149572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2506.211911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.10@tcp -m PUT -e local_timeout -r 1 [ 2506.272803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2506.326277] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2506.328035] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2506.555853] LNetError: 118555:0:(lib-move.c:4947:lnet_parse()) 192.168.204.10@tcp, src 192.168.204.10@tcp: Dropping PUT (error -108 looking up sender) [ 2508.639250] LNet: Removed LNI 192.168.204.10@tcp [ 2508.655411] LNet: Removed LNI 10.1.2.3@tcp [ 2509.030599] Key type .llcrypt unregistered [ 2509.033466] Key type ._llcrypt unregistered [ 2519.893485] Key type ._llcrypt registered [ 2519.895772] Key type .llcrypt registered [ 2521.154413] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:15:16 (1770686116) [ 2521.569799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.588543] alg: No test for adler32 (adler32-zlib) [ 2522.535532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.882058] LNet: 119863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.483641] Key type .llcrypt unregistered [ 2523.486124] Key type ._llcrypt unregistered [ 2530.626310] Key type ._llcrypt registered [ 2530.628919] Key type .llcrypt registered [ 2530.802306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2530.821852] alg: No test for adler32 (adler32-zlib) [ 2531.850622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2531.906697] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2539.677632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 2539.913284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2540.665174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.110@tcp [ 2541.059338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2541.097855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2541.133926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2541.155018] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.110@tcp [ 2541.188904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2541.232946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2541.275592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2541.303973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2541.331530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2541.357839] LNet: 120309:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2541.382488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2541.404458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2541.431454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2541.471249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2541.498451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2541.521275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2542.274715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2542.545617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2542.591326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2542.648145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2542.689936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2542.737859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2542.767546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2542.807709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2542.876980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2542.919126] LNet: 120427:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2542.953416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2543.030735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2543.123398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2543.165975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2543.218910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2543.269509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2543.974634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.110@tcp [ 2544.188191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2544.219147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2544.255618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2544.289766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2544.326114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2544.369111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2544.425683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2544.481822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2544.510079] LNet: 120544:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2544.531220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.574906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.617173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.649154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2544.678122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.725667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2545.343797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2545.662580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2545.695573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2545.740366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2545.785755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2545.836661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2545.878957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2545.912218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2545.946748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2545.989310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2546.043562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2546.077265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2546.122661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2546.152910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.181737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2546.441074] LNet: 120706:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.446505] LNetError: 120706:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.466074] LNet: Removed LNI 192.168.204.10@tcp [ 2546.786207] Key type .llcrypt unregistered [ 2546.790153] Key type ._llcrypt unregistered [ 2562.422927] Key type ._llcrypt registered [ 2562.426610] Key type .llcrypt registered [ 2563.609480] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:15:58 (1770686158) [ 2564.050866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2564.060846] alg: No test for adler32 (adler32-zlib) [ 2565.022771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2565.405390] LNet: 121600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.977531] Key type .llcrypt unregistered [ 2565.983158] Key type ._llcrypt unregistered [ 2571.799383] Key type ._llcrypt registered [ 2571.802764] Key type .llcrypt registered [ 2572.032425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2572.049602] alg: No test for adler32 (adler32-zlib) [ 2572.973408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2573.007268] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2579.330393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.110@tcp [ 2587.038358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2587.078082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2587.099329] LNet: Added LNI 192.168.204.10@tcp1 [8/256/0/180] [ 2587.784221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2588.326403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.110@tcp [ 2588.385456] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.110@tcp [ 2588.534477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2588.566875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2588.597270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2588.628770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2588.658274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2588.699232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2588.745483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2588.795823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2588.817319] LNet: 122248:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2588.839259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2588.878433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2588.919789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2588.989037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2589.049758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2589.111646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2589.888836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2590.115809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2590.139301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2590.162367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2590.194045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2590.221586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2590.257221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2590.320390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2590.375493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2590.403849] LNet: 122365:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2590.424154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2590.473386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2590.512519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2590.542605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2590.581744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2590.618838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2591.506363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.110@tcp [ 2591.750729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2591.771931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2591.798833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2591.828540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2591.855072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2591.883528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2591.914936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2591.941653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2591.969344] LNet: 122482:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2591.984481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2592.016602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2592.035512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2592.067081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2592.095965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2592.131578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2592.870243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2593.151448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2593.176637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2593.200103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2593.220757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2593.242730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2593.264908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2593.292292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2593.322136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2593.382765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2593.426220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2593.469338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2593.525903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2593.545895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2593.572083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2598.070218] LNet: 122680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2598.077931] LNetError: 122680:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2598.087573] LNet: Removed LNI 192.168.204.10@tcp [ 2598.100887] LNet: Removed LNI 192.168.204.10@tcp1 [ 2598.514793] Key type .llcrypt unregistered [ 2598.516239] Key type ._llcrypt unregistered [ 2614.281299] Key type ._llcrypt registered [ 2614.283308] Key type .llcrypt registered [ 2615.976334] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:16:50 (1770686210) [ 2622.964654] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:16:58 (1770686218) [ 2629.543774] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:17:04 (1770686224) [ 2635.654313] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:17:10 (1770686230) [ 2642.426593] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:17:17 (1770686237) [ 2649.498664] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:17:24 (1770686244) [ 2655.543364] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:17:30 (1770686250) [ 2661.740170] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:17:36 (1770686256) [ 2667.661986] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:17:42 (1770686262) [ 2673.440225] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:17:48 (1770686268) [ 2679.344387] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:17:54 (1770686274) [ 2686.582742] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:18:01 (1770686281) [ 2692.655952] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:18:07 (1770686287) [ 2699.119300] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:18:14 (1770686294) [ 2707.500501] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:18:22 (1770686302) [ 2714.677216] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:18:29 (1770686309) [ 2721.554858] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:18:36 (1770686316) [ 2727.723631] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:18:43 (1770686323) [ 2728.453494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2728.474381] alg: No test for adler32 (adler32-zlib) [ 2729.381382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2729.488921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2729.519822] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2730.493614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2730.505244] LNet: 133386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2730.509856] LNetError: 133386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2730.521382] LNet: Removed LNI 192.168.204.10@tcp [ 2730.546923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2730.609746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2730.636455] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2736.561665] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:18:51 (1770686331) [ 2742.158368] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:18:57 (1770686337) [ 2748.500855] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:19:03 (1770686343) [ 2755.195822] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:19:10 (1770686350) [ 2761.914202] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:19:16 (1770686356) [ 2769.782773] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:19:24 (1770686364) [ 2777.507318] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:19:32 (1770686372) [ 2784.101263] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:19:39 (1770686379) [ 2789.937890] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:19:45 (1770686385) [ 2796.612818] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:19:51 (1770686391) [ 2802.163181] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:19:57 (1770686397) [ 2808.885594] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:20:03 (1770686403) [ 2815.158795] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2513 sec ========== 20:20:10 (1770686410) [ 2815.365246] LNet: 140850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2815.368864] LNetError: 140850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2815.378817] LNet: Removed LNI 192.168.204.10@tcp [ 2816.032367] Key type .llcrypt unregistered [ 2816.035313] Key type ._llcrypt unregistered [ 2827.754254] Key type ._llcrypt registered [ 2827.760034] Key type .llcrypt registered [ 2828.241444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2828.251645] alg: No test for adler32 (adler32-zlib) [ 2829.345458] Lustre: Lustre: Build Version: 2.17.50_65_g69bb343 [ 2829.538574] LNet: Added LNI 192.168.204.10@tcp [8/256/0/180] [ 2831.247150] Key type lgssc registered [ 2832.386539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2881.156079] Lustre: Mounted lustre-client [ 2885.628114] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2896.483109] LustreError: 143300:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ec747f3e000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2896.495566] LustreError: 143300:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2896.517831] Lustre: Unmounted lustre-client [ 2914.701093] Key type lgssc unregistered [ 2914.970361] LNet: 143780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.985286] LNetError: 143780:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2916.007865] LNet: Removed LNI 192.168.204.10@tcp [ 2916.694150] Key type .llcrypt unregistered [ 2916.698115] Key type ._llcrypt unregistered