[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491850098 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001019] APIC: Switch to symmetric I/O mode setup [ 0.003423] x2apic enabled [ 0.004020] Switched APIC routing to physical x2apic. [ 0.005031] kvm-guest: setup PV IPIs [ 0.009962] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010034] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.012023] pid_max: default: 32768 minimum: 301 [ 0.013280] LSM: Security Framework initializing [ 0.014121] Yama: becoming mindful. [ 0.015099] SELinux: Initializing. [ 0.016174] *** VALIDATE selinux *** [ 0.025126] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.030489] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.031190] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033108] *** VALIDATE tmpfs *** [ 0.035174] *** VALIDATE proc *** [ 0.036328] *** VALIDATE cgroup *** [ 0.037015] *** VALIDATE cgroup2 *** [ 0.038299] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039145] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040020] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041028] Spectre V2 : User space: Vulnerable [ 0.042020] Speculative Store Bypass: Vulnerable [ 0.044890] debug: unmapping init [mem 0xffffffff8e259000-0xffffffff8e260fff] [ 0.046185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047694] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048032] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051017] ... value mask: 0000ffffffffffff [ 0.052022] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055286] rcu: Hierarchical SRCU implementation. [ 0.057704] smp: Bringing up secondary CPUs ... [ 0.058580] x86: Booting SMP configuration: [ 0.059026] .... node #0, CPUs: #1 #2 #3 [ 0.063102] smp: Brought up 1 node, 4 CPUs [ 0.065015] smpboot: Max logical packages: 1 [ 0.066024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152018] node 0 deferred pages initialised in 82ms [ 0.156008] devtmpfs: initialized [ 0.157187] x86/mm: Memory block size: 128MB [ 0.160000] gcov: version magic: 0x41383552 [ 0.162262] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.166116] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.168527] pinctrl core: initialized pinctrl subsystem [ 0.170290] [ 0.170776] ************************************************************* [ 0.173017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175015] ** ** [ 0.178017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179016] ** ** [ 0.181022] ** This means that this kernel is built to expose internal ** [ 0.184020] ** IOMMU data structures, which may compromise security on ** [ 0.185012] ** your system. ** [ 0.188017] ** ** [ 0.190023] ** If you see this message and you are not debugging the ** [ 0.192022] ** kernel, report this immediately to your vendor! ** [ 0.195026] ** ** [ 0.198022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200013] ************************************************************* [ 0.203008] NET: Registered protocol family 16 [ 0.205508] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208104] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.211122] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215207] cpuidle: using governor menu [ 0.217712] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.220680] PCI: Using configuration type 1 for base access [ 0.222166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.231136] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.232061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.235195] cryptd: max_cpu_qlen set to 1000 [ 0.238295] ACPI: Added _OSI(Module Device) [ 0.239022] ACPI: Added _OSI(Processor Device) [ 0.240021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.242019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245571] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252460] ACPI: Interpreter enabled [ 0.253090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.254018] ACPI: Using IOAPIC for interrupt routing [ 0.256203] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.259505] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.272059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.274032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.278116] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.282235] acpiphp: Slot [2] registered [ 0.284179] acpiphp: Slot [3] registered [ 0.285000] acpiphp: Slot [4] registered [ 0.285000] acpiphp: Slot [5] registered [ 0.287250] acpiphp: Slot [6] registered [ 0.289145] acpiphp: Slot [7] registered [ 0.290136] acpiphp: Slot [8] registered [ 0.291129] acpiphp: Slot [9] registered [ 0.292072] acpiphp: Slot [10] registered [ 0.294126] acpiphp: Slot [11] registered [ 0.296193] acpiphp: Slot [12] registered [ 0.297127] acpiphp: Slot [13] registered [ 0.297888] acpiphp: Slot [14] registered [ 0.299076] acpiphp: Slot [15] registered [ 0.300139] acpiphp: Slot [16] registered [ 0.301106] acpiphp: Slot [17] registered [ 0.302087] acpiphp: Slot [18] registered [ 0.304144] acpiphp: Slot [19] registered [ 0.305117] acpiphp: Slot [20] registered [ 0.307143] acpiphp: Slot [21] registered [ 0.309142] acpiphp: Slot [22] registered [ 0.310201] acpiphp: Slot [23] registered [ 0.312173] acpiphp: Slot [24] registered [ 0.313132] acpiphp: Slot [25] registered [ 0.315131] acpiphp: Slot [26] registered [ 0.316157] acpiphp: Slot [27] registered [ 0.318103] acpiphp: Slot [28] registered [ 0.319125] acpiphp: Slot [29] registered [ 0.320120] acpiphp: Slot [30] registered [ 0.322144] acpiphp: Slot [31] registered [ 0.323108] PCI host bridge to bus 0000:00 [ 0.324026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.326035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.328026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.332034] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.335039] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.341568] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.346792] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354018] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.359073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.361021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.363029] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.369336] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.371799] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.374068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.377497] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.380015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.390030] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.395021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.399353] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.411018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421019] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.428108] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.433015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.436015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.453031] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.463385] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.467539] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.470527] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.473497] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.475300] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.479217] iommu: Default domain type: Passthrough [ 0.481594] SCSI subsystem initialized [ 0.483208] ACPI: bus type USB registered [ 0.484220] usbcore: registered new interface driver usbfs [ 0.487115] usbcore: registered new interface driver hub [ 0.488129] usbcore: registered new device driver usb [ 0.490232] pps_core: LinuxPPS API ver. 1 registered [ 0.492023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.495126] PTP clock support registered [ 0.498130] EDAC MC: Ver: 3.0.0 [ 0.500164] PCI: Using ACPI for IRQ routing [ 0.502900] NetLabel: Initializing [ 0.503009] NetLabel: domain hash size = 128 [ 0.505014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.506111] NetLabel: unlabeled traffic allowed by default [ 0.509066] vgaarb: loaded [ 0.510353] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.512017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.520407] clocksource: Switched to clocksource kvm-clock [ 0.628285] VFS: Disk quotas dquot_6.6.0 [ 0.630029] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.632717] *** VALIDATE ramfs *** [ 0.634145] *** VALIDATE hugetlbfs *** [ 0.636015] pnp: PnP ACPI init [ 0.638709] pnp: PnP ACPI: found 6 devices [ 0.658893] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.661696] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.663522] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.665786] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.668472] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.671253] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.674399] NET: Registered protocol family 2 [ 0.677204] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.682419] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.686380] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.692153] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.696164] TCP: Hash tables configured (established 65536 bind 65536) [ 0.699420] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.703187] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.706222] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.708955] NET: Registered protocol family 1 [ 0.712171] RPC: Registered named UNIX socket transport module. [ 0.714602] RPC: Registered udp transport module. [ 0.716489] RPC: Registered tcp transport module. [ 0.718398] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.720596] NET: Registered protocol family 44 [ 0.722534] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.724404] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.726467] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.729029] PCI: CLS 0 bytes, default 64 [ 0.730926] Unpacking initramfs... [ 2.150266] debug: unmapping init [mem 0xffff8da83cc64000-0xffff8da83ffcffff] [ 2.154834] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.157217] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.160331] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.659628] Initialise system trusted keyrings [ 2.661324] Key type blacklist registered [ 2.663354] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.672537] zbud: loaded [ 2.675543] *** VALIDATE nfs *** [ 2.676856] *** VALIDATE nfs4 *** [ 2.678649] pstore: using deflate compression [ 2.682087] Platform Keyring initialized [ 2.781843] NET: Registered protocol family 38 [ 2.783946] Key type asymmetric registered [ 2.785641] Asymmetric key parser 'x509' registered [ 2.787377] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.790471] io scheduler mq-deadline registered [ 2.791968] io scheduler kyber registered [ 2.793387] io scheduler bfq registered [ 2.795332] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.798461] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.801410] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.804547] ACPI: Power Button [PWRF] [ 2.896613] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.986456] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.082220] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.112492] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.140964] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.146661] Non-volatile memory driver v1.3 [ 3.148178] Linux agpgart interface v0.103 [ 3.179682] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.182252] vda: detected capacity change from 0 to 68624384 [ 3.197653] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.200284] vdb: detected capacity change from 0 to 1073741824 [ 3.206178] libphy: Fixed MDIO Bus: probed [ 3.212527] usbcore: registered new interface driver usbserial_generic [ 3.219116] usbserial: USB Serial support registered for generic [ 3.223401] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.228081] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.229868] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.236221] mousedev: PS/2 mouse device common for all mice [ 3.243107] rtc_cmos 00:05: RTC can wake from S4 [ 3.245258] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.251147] rtc_cmos 00:05: registered as rtc0 [ 3.251367] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.252952] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.260406] intel_pstate: CPU model not supported [ 3.262912] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.267198] hid: raw HID events driver (C) Jiri Kosina [ 3.268964] usbcore: registered new interface driver usbhid [ 3.270598] usbhid: USB HID core driver [ 3.272336] drop_monitor: Initializing network drop monitor service [ 3.274669] Initializing XFRM netlink socket [ 3.276621] NET: Registered protocol family 10 [ 3.278979] Segment Routing with IPv6 [ 3.280725] NET: Registered protocol family 17 [ 3.282950] mpls_gso: MPLS GSO support [ 3.290425] RAS: Correctable Errors collector initialized. [ 3.293098] AVX version of gcm_enc/dec engaged. [ 3.294290] AES CTR mode by8 optimization enabled [ 3.376323] sched_clock: Marking stable (3376276474, 0)->(4434663068, -1058386594) [ 3.380660] registered taskstats version 1 [ 3.383050] Loading compiled-in X.509 certificates [ 3.384992] zswap: loaded using pool lzo/zbud [ 3.413656] Key type big_key registered [ 3.426935] Key type encrypted registered [ 3.428652] ima: No TPM chip found, activating TPM-bypass! [ 3.430747] ima: Allocated hash algorithm: sha1 [ 3.432401] ima: No architecture policies found [ 3.434223] evm: Initialising EVM extended attributes: [ 3.436124] evm: security.selinux [ 3.437260] evm: security.ima [ 3.437960] evm: security.capability [ 3.438853] evm: HMAC attrs: 0x1 [ 3.440965] rtc_cmos 00:05: setting system clock to 2025-12-22 14:08:10 UTC (1766412490) [ 3.447386] debug: unmapping init [mem 0xffffffff8f203000-0xffffffff8f3fffff] [ 3.450619] debug: unmapping init [mem 0xffffffff8df82000-0xffffffff8e258fff] [ 3.460127] Write protecting the kernel read-only data: 28672k [ 3.463036] debug: unmapping init [mem 0xffffffff8c603000-0xffffffff8c7fffff] [ 3.465857] debug: unmapping init [mem 0xffffffff8cf14000-0xffffffff8cffffff] [ 3.498701] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.505970] systemd[1]: Detected virtualization kvm. [ 3.507689] systemd[1]: Detected architecture x86-64. [ 3.509364] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.537735] systemd[1]: No hostname configured. [ 3.539661] systemd[1]: Set hostname to . [ 3.541882] random: systemd: uninitialized urandom read (16 bytes read) [ 3.544175] systemd[1]: Initializing machine ID from random generator. [ 3.600193] random: ln: uninitialized urandom read (6 bytes read) [ 3.703283] random: systemd: uninitialized urandom read (16 bytes read) [ 3.705915] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.709841] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.714578] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.376792] device-mapper: uevent: version 1.0.3 [ 4.379312] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.181876] virtio_net virtio0 ens2: renamed from eth0 [ 5.208255] scsi host0: ata_piix [ 5.232404] scsi host1: ata_piix [ 5.235543] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.238225] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.582708] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.944362] random: crng init done [ 9.945752] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.421574] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.486310] printk: systemd: 25 output lines suppressed due to ratelimiting [ 13.401797] SELinux: Disabled at runtime. [ 13.557205] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.579341] systemd[1]: Detected virtualization kvm. [ 13.580862] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 14.951586] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.962608] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.981458] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.992768] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.003290] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.019419] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.048947] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ 15.298202] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 16.493918] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.414895] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.805240] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.898326] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 18.037702] EDAC sbridge: Ver: 1.1.2 [ 21.944940] Key type dns_resolver registered [ 22.136252] hrtimer: interrupt took 5353431 ns [* ] A start job is running for Configur…-only root support (7s / no limit)[ 22.530773] NFS: Registering the id_resolver key type [ 22.532534] Key type id_resolver registered [ 22.534110] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg436-client login: [ 80.432147] libcfs: loading out-of-tree module taints kernel. [ 80.496127] Key type ._llcrypt registered [ 80.507216] Key type .llcrypt registered [ 81.299817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 81.311847] alg: No test for adler32 (adler32-zlib) [ 82.612068] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 83.626519] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 85.407204] Key type lgssc registered [ 86.819051] Lustre: Echo OBD driver; http://www.lustre.org/ [ 192.787443] Lustre: Mounted lustre-client [ 197.186245] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 210.999308] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing check_logdir /tmp/testlogs/ [ 214.617551] Lustre: DEBUG MARKER: oleg436-client.virtnet: executing yml_node [ 218.282848] Lustre: DEBUG MARKER: Client: 2.17.0.RC3 [ 218.591541] Lustre: lustre-OST0000-osc-ffff8da8872c9800: disconnect after 23s idle [ 220.160846] Lustre: DEBUG MARKER: MDS: 2.17.0.RC3 [ 222.044855] Lustre: DEBUG MARKER: OSS: 2.17.0.RC3 [ 223.330738] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Dec 22 09:11:49 EST 2025 [ 236.203203] LustreError: 7890:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8da8872c9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 236.224830] LustreError: 7890:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 236.259593] Lustre: Unmounted lustre-client [ 268.715559] Key type lgssc unregistered [ 269.073227] LNet: 8339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 269.084576] LNetError: 8339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 269.109410] LNet: Removed LNI 192.168.204.36@tcp [ 270.025536] Key type .llcrypt unregistered [ 270.028728] Key type ._llcrypt unregistered [ 288.007548] Key type ._llcrypt registered [ 288.009303] Key type .llcrypt registered [ 288.214520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 288.227333] alg: No test for adler32 (adler32-zlib) [ 289.102787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 289.192904] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 289.227692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 292.497170] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 292.580206] LNet: 9203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.586939] LNetError: 9203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 292.610630] LNet: Removed LNI 192.168.204.36@tcp [ 293.178621] Key type .llcrypt unregistered [ 293.180500] Key type ._llcrypt unregistered [ 297.293513] Key type ._llcrypt registered [ 297.295220] Key type .llcrypt registered [ 298.285490] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:13:04 (1766412784) [ 298.502873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.515272] alg: No test for adler32 (adler32-zlib) [ 299.395365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.484568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.499700] LNet: 9618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.523613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.573786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 299.586250] LNet: 9625:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 305.194734] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:13:10 (1766412790) [ 305.350126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.374268] LNet: 10201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.420210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.488545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 311.637882] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:13:17 (1766412797) [ 311.749660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.780120] LNet: 10786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.808394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.877737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 317.200961] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:13:22 (1766412802) [ 317.246027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 317.259320] LNet: 11363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 317.286135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 317.337916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 322.305988] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:13:28 (1766412808) [ 322.370978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.405879] LNet: 11948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.436394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.523853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 322.838140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 327.659844] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:13:33 (1766412813) [ 327.756401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.794817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.858247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 328.149877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 328.426340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 332.909821] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:13:39 (1766412819) [ 332.982194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 333.003279] LNet: 13134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 333.007255] LNet: 13134:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 333.032109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 333.079413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 338.162536] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:13:44 (1766412824) [ 338.275718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.331455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.378434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 338.416944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 338.695841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 338.720824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 339.227932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 339.265551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 339.583825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 339.632397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 339.960183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 339.981932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 340.335720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 340.365686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 340.834773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 340.882685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 341.299241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 341.327170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 341.678992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 341.731041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 342.121550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 342.156565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 342.524169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 342.551220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 342.914456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 342.955983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 343.330577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 343.364604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 348.826827] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:13:54 (1766412834) [ 348.893553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.934804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.017794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 349.040841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 354.887732] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:14:00 (1766412840) [ 355.002115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.029912] LNet: 14990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.036511] LNet: 14990:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 355.083429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.178453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 355.209686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 361.332524] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:14:07 (1766412847) [ 361.461587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.514554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.595565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 361.622155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 366.464427] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:14:12 (1766412852) [ 366.538602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.573694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.624792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 366.645963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 370.891122] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:14:16 (1766412856) [ 370.969751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.009858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.089275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 371.140396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 376.071530] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:14:22 (1766412862) [ 376.139590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.168457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.257271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 376.294941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 381.531750] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:14:27 (1766412867) [ 381.579828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.607333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.673861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 381.948587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 387.019268] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:14:32 (1766412872) [ 387.227350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.243237] LNet: 18520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 387.245482] LNet: 18520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 387.268077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.410217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 387.636673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 393.206762] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:14:39 (1766412879) [ 393.409504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.444091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.570211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 393.762938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 398.971846] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:14:44 (1766412884) [ 399.079844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.144844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.596541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 399.831166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 406.209114] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:14:51 (1766412891) [ 406.364250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.429383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.488241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.532896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 413.036488] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:14:58 (1766412898) [ 413.210250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.279848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.371677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 413.454764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.738560] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:15:05 (1766412905) [ 419.824849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.903656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.988178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 425.506465] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:15:11 (1766412911) [ 425.606907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.644795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.715903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 431.892949] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:15:17 (1766412917) [ 431.945773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.009832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.121229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 432.537028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 437.885821] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:15:23 (1766412923) [ 437.944435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 437.998546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.144493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 438.172484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 444.169917] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:15:29 (1766412929) [ 444.238871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.270750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.360495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 444.384199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 450.416374] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:15:36 (1766412936) [ 450.492059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.515762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.578420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 450.613880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 456.535847] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:15:42 (1766412942) [ 456.607675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.635124] LNet: 25118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 456.638587] LNet: 25118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 456.664051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.750769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 456.777971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 463.023688] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:15:48 (1766412948) [ 463.117505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.162464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.249229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 463.344483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 463.367854] LNetError: 25722:0:(peer.c:2225:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 463.427476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 463.475735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 469.605500] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:15:55 (1766412955) [ 469.718326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.827337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.950856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SmGjalRSmZ [ 470.036328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SmGjalRSmZ [ 470.079244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.SmGjalRSmZ [ 476.135669] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:16:01 (1766412961) [ 476.194665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.216093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.266150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 476.288528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 476.322123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 482.052968] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:16:07 (1766412967) [ 482.158380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 482.196617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 482.263848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 483.965291] LNet: Message forwarding will be enabled in 2s [ 487.071202] LNet: Message forwarding enabled [ 487.404921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 487.428156] LNet: Message forwarding will stop in 2s to 6s [ 491.892627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 492.377854] LNet: Message forwarding disabled [ 493.537747] LNet: Message forwarding will be enabled in 2s [ 497.503170] LNet: Message forwarding enabled [ 498.509625] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:16:24 (1766412984) [ 498.621278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.598710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.662481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 504.505419] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:16:30 (1766412990) [ 504.622585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.662989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.730649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 504.753441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 504.781898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 504.814109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 504.840222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 504.870244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 504.891946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 504.937641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 504.964661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 504.998357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 505.026719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 505.047325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 505.072911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 505.119148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 505.170426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 505.205383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 505.246492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 505.289458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 505.325196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 505.340969] LNetError: 28791:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 510.578464] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:16:36 (1766412996) [ 510.892115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.941782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.034972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.310844] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:16:42 (1766413002) [ 519.136812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg436-server [ 520.823460] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:16:46 (1766413006) [ 520.884827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.939205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.995050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 521.017604] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 522.470422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[37-41]@tcp [ 522.501207] LNet: 30506:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 522.566528] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.40@tcp is being used as a gateway but routing feature is not turned on [ 522.574944] LNetError: 30292:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.40@tcp has gone from up to down [ 523.566127] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.41@tcp is being used as a gateway but routing feature is not turned on [ 523.571612] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 524.586151] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.40@tcp is being used as a gateway but routing feature is not turned on [ 524.600652] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 526.638868] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.39@tcp is being used as a gateway but routing feature is not turned on [ 526.660881] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 528.423202] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:16:54 (1766413014) [ 530.103348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg436-server [ 530.725429] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.38@tcp is being used as a gateway but routing feature is not turned on [ 530.734241] LNetError: 30292:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 531.808333] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:16:57 (1766413017) [ 532.030666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.067575] LNetError: 31441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.085222] LNet: Removed LNI 192.168.204.36@tcp [ 532.109398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.187575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 532.236476] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 532.831837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[37-42/2]@tcp [ 532.874879] LNet: 31558:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 532.880169] LNet: 31558:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 532.897153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[37-42/2]@tcp [ 532.906142] LNetError: 31445:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.37@tcp has gone from up to down [ 532.917132] LNetError: 31445:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 538.257838] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:17:04 (1766413024) [ 538.444435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.475240] LNetError: 32150:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 538.508171] LNet: Removed LNI 192.168.204.36@tcp [ 538.551357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.664900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 538.752541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.789876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.871924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 538.920399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.960130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.995042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 539.064839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.099186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.161558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.239795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.275610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.340466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 539.414570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.446699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.516410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.586360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.616663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.666977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 539.724362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.763471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.814604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.878263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.988263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.154255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 540.209930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.238542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.296567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.359330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.397964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.445865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 540.493763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.534230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.585738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 545.780523] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:17:11 (1766413031) [ 547.475525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg436-server [ 548.990982] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:17:14 (1766413034) [ 550.409277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg436-server [ 552.037485] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:17:17 (1766413037) [ 552.617972] Key type .llcrypt unregistered [ 552.621490] Key type ._llcrypt unregistered [ 557.322818] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 557.500616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 557.503549] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 557.506214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 559.935342] Key type ._llcrypt registered [ 559.937049] Key type .llcrypt registered [ 560.093486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 560.108785] alg: No test for adler32 (adler32-zlib) [ 561.001338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.040433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 561.061237] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 561.681973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 561.697821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 563.847955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 563.871776] LNet: Removed LNI 10.1.2.3@tcp [ 564.040981] LNet: 34261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.050519] LNetError: 34261:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.059828] LNet: Removed LNI 192.168.204.36@tcp [ 564.816716] Key type .llcrypt unregistered [ 564.822779] Key type ._llcrypt unregistered [ 574.914279] Key type ._llcrypt registered [ 574.915934] Key type .llcrypt registered [ 576.263421] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:17:41 (1766413061) [ 576.731286] Key type .llcrypt unregistered [ 576.733579] Key type ._llcrypt unregistered [ 581.590413] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 581.758802] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 581.761412] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 581.767836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 584.315188] Key type ._llcrypt registered [ 584.316833] Key type .llcrypt registered [ 584.474533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.495219] alg: No test for adler32 (adler32-zlib) [ 585.427068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.500106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.567751] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 586.468202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 586.510832] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 589.320570] LNet: 35746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.326749] LNetError: 35746:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.334330] LNet: Removed LNI 192.168.204.36@tcp [ 589.344615] LNet: Removed LNI 10.1.2.3@tcp [ 590.078854] Key type .llcrypt unregistered [ 590.080772] Key type ._llcrypt unregistered [ 603.119268] Key type ._llcrypt registered [ 603.120685] Key type .llcrypt registered [ 604.469436] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:18:10 (1766413090) [ 604.931324] Key type .llcrypt unregistered [ 604.936509] Key type ._llcrypt unregistered [ 610.883857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 611.042877] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 611.049389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 611.051753] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 613.530940] Key type ._llcrypt registered [ 613.532530] Key type .llcrypt registered [ 613.697701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 613.716959] alg: No test for adler32 (adler32-zlib) [ 614.641584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.683619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 614.705707] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 616.890144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 616.914768] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 618.788398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 618.814429] LNet: Removed LNI 10.1.2.3@tcp [ 618.830315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 618.863362] LNet: Removed LNI 10.1.2.31@tcp [ 618.890267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 618.914664] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.648163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 619.666783] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 620.575386] LNet: 37654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 620.584376] LNetError: 37654:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 620.592459] LNet: Removed LNI 10.1.2.3@tcp [ 620.606122] LNet: Removed LNI 10.1.2.31@tcp [ 621.094574] Key type .llcrypt unregistered [ 621.096221] Key type ._llcrypt unregistered [ 632.170748] Key type ._llcrypt registered [ 632.171979] Key type .llcrypt registered [ 633.679165] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:18:39 (1766413119) [ 634.109111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 634.152675] alg: No test for adler32 (adler32-zlib) [ 635.150132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.244641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.284209] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 636.479909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.497316] LNet: 38603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.501455] LNetError: 38603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.518323] LNet: Removed LNI 192.168.204.36@tcp [ 636.566114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.629578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.36@tcp [ 636.654814] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 637.633689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.36@tcp [ 637.688509] LNet: Removed LNI 192.168.204.36@tcp [ 637.763727] LNet: 38738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.442568] Key type .llcrypt unregistered [ 638.444382] Key type ._llcrypt unregistered [ 648.384465] Key type ._llcrypt registered [ 648.386023] Key type .llcrypt registered [ 649.712874] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:18:55 (1766413135) [ 651.376174] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg436-server [ 653.497964] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:18:58 (1766413138) [ 654.036405] Key type .llcrypt unregistered [ 654.038817] Key type ._llcrypt unregistered [ 661.401242] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 661.652328] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 661.654714] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 661.670724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 664.332610] Key type ._llcrypt registered [ 664.336562] Key type .llcrypt registered [ 664.491705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.507640] alg: No test for adler32 (adler32-zlib) [ 665.443992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.505054] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 666.262345] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 668.524884] LNet: 40566:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.533126] LNetError: 40566:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.542175] LNet: Removed LNI 192.168.204.36@tcp1 [ 668.551850] LNet: Removed LNI 10.1.2.3@tcp2 [ 669.157580] Key type .llcrypt unregistered [ 669.162657] Key type ._llcrypt unregistered [ 679.606068] Key type ._llcrypt registered [ 679.607656] Key type .llcrypt registered [ 680.793292] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:19:26 (1766413166) [ 681.167880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 681.185600] alg: No test for adler32 (adler32-zlib) [ 682.120405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.198535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.227332] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 683.314734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.340409] LNet: 41531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.343982] LNetError: 41531:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.356298] LNet: Removed LNI 192.168.204.36@tcp [ 683.388824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.457941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 683.487810] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 689.746816] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:19:35 (1766413175) [ 690.054595] LNet: 42123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.070408] LNetError: 42123:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 690.083657] LNet: Removed LNI 192.168.204.36@tcp [ 690.911487] Key type .llcrypt unregistered [ 690.913290] Key type ._llcrypt unregistered [ 696.441043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 696.631042] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 696.633738] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 696.657689] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.236585] Key type ._llcrypt registered [ 699.238023] Key type .llcrypt registered [ 699.457643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.466464] alg: No test for adler32 (adler32-zlib) [ 700.430437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.476205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.501383] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 701.442977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 701.471386] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.398718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.430752] LNet: 42857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.439807] LNetError: 42857:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.449136] LNet: Removed LNI 192.168.204.36@tcp [ 703.473265] LNet: Removed LNI 10.1.2.3@tcp [ 703.516452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.594523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 703.630920] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 703.634548] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.317144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 704.345113] LNet: 42884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.351353] LNetError: 42884:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 704.355635] LNetError: 42884:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 704.366700] LNet: Removed LNI 192.168.204.36@tcp [ 704.382169] LNet: Removed LNI 10.1.2.3@tcp [ 704.412277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.496759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 704.531767] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 704.550463] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.879380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 704.893874] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 705.857829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.893221] LNet: 43046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.907762] LNetError: 43046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.911021] LNetError: 43046:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 705.922837] LNet: Removed LNI 192.168.204.36@tcp [ 705.925491] LNet: Removed LNI 10.1.2.3@tcp [ 705.947478] LNet: Removed LNI 192.168.204.36@tcp2 [ 705.975576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.060684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 706.095932] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 706.102062] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 706.105180] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 711.777520] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:19:57 (1766413197) [ 711.971293] LNet: 43643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.982118] LNetError: 43643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.986701] LNetError: 43643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 711.992389] LNet: Removed LNI 192.168.204.36@tcp [ 711.994280] LNet: Removed LNI 10.1.2.3@tcp [ 712.011076] LNet: Removed LNI 192.168.204.36@tcp2 [ 712.584243] Key type .llcrypt unregistered [ 712.586040] Key type ._llcrypt unregistered [ 717.218656] Key type ._llcrypt registered [ 717.220297] Key type .llcrypt registered [ 717.451353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.467153] alg: No test for adler32 (adler32-zlib) [ 718.387779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.444654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.487294] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 719.381471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.403385] LNet: 44033:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.414389] LNetError: 44033:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.428214] LNet: Removed LNI 192.168.204.36@tcp [ 719.446089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.506338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 719.527978] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 719.947808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.968636] LNet: 44063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.973385] LNetError: 44063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.989264] LNet: Removed LNI 192.168.204.36@tcp [ 720.003304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.070601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 720.098068] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 720.422157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.458276] LNet: Removed LNI 192.168.204.36@tcp [ 720.475111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.532577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 720.555482] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 720.926073] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 720.980965] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 720.991730] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 720.993768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.340302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 723.364682] LNet: 44172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 723.369175] LNet: 44172:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 723.374710] LNetError: 44172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 723.378097] LNetError: 44172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 723.390884] LNet: Removed LNI 192.168.204.36@tcp [ 723.418623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 723.471947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 723.502144] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 729.361384] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:20:15 (1766413215) [ 729.510079] LNet: 44761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 729.529489] LNetError: 44761:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.549380] LNet: Removed LNI 192.168.204.36@tcp [ 730.255915] Key type .llcrypt unregistered [ 730.257562] Key type ._llcrypt unregistered [ 735.643925] Key type ._llcrypt registered [ 735.647524] Key type .llcrypt registered [ 735.900411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 735.919883] alg: No test for adler32 (adler32-zlib) [ 736.797266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.858461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 736.887257] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 737.949149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.980428] LNet: 45163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.003934] LNetError: 45163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.019947] LNet: Removed LNI 192.168.204.36@tcp [ 738.055187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.174203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 738.213529] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 745.160294] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:20:30 (1766413230) [ 745.406474] LNet: 45755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.418602] LNetError: 45755:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.432808] LNet: Removed LNI 192.168.204.36@tcp [ 746.046686] Key type .llcrypt unregistered [ 746.048647] Key type ._llcrypt unregistered [ 751.024382] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 751.121087] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 751.127990] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 751.142144] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.784945] Key type ._llcrypt registered [ 753.792344] Key type .llcrypt registered [ 754.014453] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.032766] alg: No test for adler32 (adler32-zlib) [ 754.934121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.005147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 755.040276] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 755.926670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.946817] LNet: 46200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.956396] LNetError: 46200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.972361] LNet: Removed LNI 192.168.204.36@tcp [ 755.996296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.091473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 756.117517] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 762.971460] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:20:48 (1766413248) [ 763.077515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.122332] LNet: 46792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.133605] LNetError: 46792:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.154978] LNet: Removed LNI 192.168.204.36@tcp [ 763.183996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.309718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 769.903674] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:20:55 (1766413255) [ 770.196276] LNet: 47381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.953857] Key type .llcrypt unregistered [ 770.955700] Key type ._llcrypt unregistered [ 777.318310] Key type ._llcrypt registered [ 777.320604] Key type .llcrypt registered [ 777.472778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.489630] alg: No test for adler32 (adler32-zlib) [ 778.391708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 778.454522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 778.492972] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 779.481795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.512459] LNet: 47773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.522782] LNetError: 47773:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.533631] LNet: Removed LNI 192.168.204.36@tcp [ 779.550852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.641476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 779.678921] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 785.927359] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:21:11 (1766413271) [ 786.171918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.205907] LNet: 48366:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.221813] LNetError: 48366:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.243179] LNet: Removed LNI 192.168.204.36@tcp [ 786.265456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.360429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 792.150964] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:21:17 (1766413277) [ 793.811386] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 795.499282] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:21:21 (1766413281) [ 796.968626] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 798.650517] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:21:24 (1766413284) [ 800.076897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 802.306607] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:21:27 (1766413287) [ 804.090266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 805.947075] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:21:31 (1766413291) [ 807.914935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 809.927673] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:21:35 (1766413295) [ 810.007695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.042243] LNet: 50410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.086983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.155507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 810.208675] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 811.055577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.36 [ 811.085671] LNet: 50550:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 811.090828] LNet: Removed LNI 192.168.204.36@tcp [ 816.859628] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:21:42 (1766413302) [ 817.014880] LNet: 51115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.019657] LNetError: 51115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.044333] LNet: Removed LNI 192.168.204.36@tcp [ 817.633964] Key type .llcrypt unregistered [ 817.635679] Key type ._llcrypt unregistered [ 822.191428] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 822.360726] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 822.363920] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 822.370942] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 825.011411] Key type ._llcrypt registered [ 825.013461] Key type .llcrypt registered [ 825.165931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 825.183542] alg: No test for adler32 (adler32-zlib) [ 826.535158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 832.507536] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:21:58 (1766413318) [ 832.708927] LNet: 52019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.719205] LNetError: 52019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 832.757414] LNet: Removed LNI 10.1.2.3@tcp [ 833.474776] Key type .llcrypt unregistered [ 833.479919] Key type ._llcrypt unregistered [ 838.532461] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 838.743839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 838.745901] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 838.748312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 841.520427] Key type ._llcrypt registered [ 841.526849] Key type .llcrypt registered [ 841.794935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.811401] alg: No test for adler32 (adler32-zlib) [ 842.784393] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 848.438597] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:22:14 (1766413334) [ 848.578115] LNet: 52905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 848.600459] LNetError: 52905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 848.637788] LNet: Removed LNI 10.1.2.3@tcp [ 849.485777] Key type .llcrypt unregistered [ 849.489329] Key type ._llcrypt unregistered [ 854.439558] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 854.666100] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 854.672125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 854.678690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.263624] Key type ._llcrypt registered [ 857.265320] Key type .llcrypt registered [ 857.431884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 857.446598] alg: No test for adler32 (adler32-zlib) [ 858.359355] LNet: 53207:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 858.371111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.096256] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:22:29 (1766413349) [ 864.258184] LNet: 53791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.265561] LNetError: 53791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.303291] LNet: Removed LNI 10.1.2.3@tcp [ 865.127635] Key type .llcrypt unregistered [ 865.129329] Key type ._llcrypt unregistered [ 870.559807] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 870.768827] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 870.771198] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 870.776228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 873.461377] Key type ._llcrypt registered [ 873.462944] Key type .llcrypt registered [ 873.706410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.723890] alg: No test for adler32 (adler32-zlib) [ 874.680074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.757371] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.053022] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:22:48 (1766413368) [ 883.514140] LNet: 55027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.523868] LNetError: 55027:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.540032] LNet: Removed LNI 10.1.2.3@tcp [ 884.279042] Key type .llcrypt unregistered [ 884.280621] Key type ._llcrypt unregistered [ 890.561118] Key type ._llcrypt registered [ 890.562814] Key type .llcrypt registered [ 890.833396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.856648] alg: No test for adler32 (adler32-zlib) [ 891.762856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 891.805136] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 899.985920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 900.588206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_interrupt -r 1 [ 900.643751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_interrupt -r 1 [ 900.709298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 900.736744] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 900.940800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 900.963760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 900.996562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.025357] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 901.042405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.087754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.114916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 901.148934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 901.191874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 901.210958] LNet: 55461:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 901.225025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 901.250862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 901.280994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 901.307824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 901.350854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 901.395837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 901.455878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 901.910419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_dropped -r 1 [ 901.951454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_dropped -r 1 [ 901.989841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 902.014225] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 902.018191] LNet: Skipped 1 previous similar message [ 902.280430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 902.311654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 902.337201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 902.379695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 902.404258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 902.435692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 902.474411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 902.507421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 902.521227] LNet: 55570:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.544679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 902.574314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 902.610901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 902.642498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 902.667274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 902.698106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 902.751319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 903.414683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_aborted -r 1 [ 903.467433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_aborted -r 1 [ 903.509541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 903.535740] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 903.770315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 903.813645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 903.872235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 903.904692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 903.946599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 904.004701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 904.064345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 904.127339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 904.143429] LNet: 55679:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 904.167811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 904.233260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 904.277954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 904.326415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 904.370188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 904.438608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 904.492697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 905.095819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_no_route -r 1 [ 905.142814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_no_route -r 1 [ 905.189194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 905.458071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 905.513693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 905.555904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 905.592309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 905.622531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 905.661656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 905.702224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 905.743739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 905.782662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 905.834119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 905.869397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 905.904728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 905.930516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 905.968510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 906.003924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 906.564338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_timeout -r 1 [ 906.597674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_timeout -r 1 [ 906.632755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 906.655488] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 906.659150] LNet: Skipped 1 previous similar message [ 906.912857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 906.956655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 906.987152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 907.021819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 907.060562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 907.088605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 907.141787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 907.174745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 907.206186] LNet: 55898:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 907.211396] LNet: 55898:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 907.237587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 907.292360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 907.330538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 907.377804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 907.418410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 907.443258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 907.493110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 908.057893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_error -r 1 [ 908.090817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_error -r 1 [ 908.127688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 908.401787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 908.439798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 908.483159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 908.513313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 908.551862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 908.581995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 908.609074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 908.635122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 908.664833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 908.692100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 908.713964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 908.766910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 908.814354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.844431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.885455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 909.126685] LNet: 56047:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.135243] LNetError: 56047:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.152944] LNet: Removed LNI 192.168.204.36@tcp [ 909.466807] Key type .llcrypt unregistered [ 909.470366] Key type ._llcrypt unregistered [ 923.106747] Key type ._llcrypt registered [ 923.109102] Key type .llcrypt registered [ 924.482640] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:23:30 (1766413410) [ 925.346503] Key type .llcrypt unregistered [ 925.348083] Key type ._llcrypt unregistered [ 930.960399] Key type ._llcrypt registered [ 930.962126] Key type .llcrypt registered [ 931.125909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.138894] alg: No test for adler32 (adler32-zlib) [ 932.102304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 932.161047] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 938.890836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 948.175814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.200650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 948.222768] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 949.842039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_interrupt -r 1 [ 949.871879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_interrupt -r 1 [ 949.917775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_interrupt -r 1 [ 949.984714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_interrupt -r 1 [ 950.046000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_interrupt -r 1 [ 950.108237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_interrupt -r 1 [ 950.155967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_interrupt -r 1 [ 950.184019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_interrupt -r 1 [ 950.204287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 950.214721] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 950.218785] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 950.475573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 950.508486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 950.532803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 950.560319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 950.595057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 950.628520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 950.664882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 950.698814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 950.723099] LNet: 57511:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 950.758050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 950.842672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 950.901087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 950.971051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.012067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.060972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.108371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 951.641424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_dropped -r 1 [ 951.666545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_dropped -r 1 [ 951.695993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_dropped -r 1 [ 951.726079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_dropped -r 1 [ 951.758231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_dropped -r 1 [ 951.784986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_dropped -r 1 [ 951.809485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_dropped -r 1 [ 951.841355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_dropped -r 1 [ 951.890816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 951.919526] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 951.927027] LNet: Skipped 2 previous similar messages [ 952.147584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.179345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 952.212877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.238633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.275911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.336502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 952.364335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 952.389879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 952.407294] LNet: 57632:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 952.426967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 952.462184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 952.490852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 952.539603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 952.565223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 952.595918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 952.621290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 953.106537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_aborted -r 1 [ 953.154508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_aborted -r 1 [ 953.191367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_aborted -r 1 [ 953.223900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_aborted -r 1 [ 953.279937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_aborted -r 1 [ 953.318800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_aborted -r 1 [ 953.359555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_aborted -r 1 [ 953.393964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_aborted -r 1 [ 953.440634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 953.474025] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 953.479990] LNet: Skipped 2 previous similar messages [ 953.722432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 953.757322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 953.817693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 953.859650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 953.892381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 953.921723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 953.952447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 953.986611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.018951] LNet: 57753:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 954.035772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.076990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.123206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.163490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.191796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.232057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.286013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 954.868312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_no_route -r 1 [ 954.911931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_no_route -r 1 [ 954.969660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_no_route -r 1 [ 955.010636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_no_route -r 1 [ 955.065081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_no_route -r 1 [ 955.094727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_no_route -r 1 [ 955.146364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_no_route -r 1 [ 955.199408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_no_route -r 1 [ 955.229842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 955.499856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 955.538060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 955.577996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 955.634803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 955.686500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 955.724949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 955.775757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 955.823506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 955.863586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 955.898367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 955.933299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.966430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 955.994719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 956.028648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 956.057085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 956.632396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_timeout -r 1 [ 956.676488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_timeout -r 1 [ 956.752730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_timeout -r 1 [ 956.795473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_timeout -r 1 [ 956.856300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_timeout -r 1 [ 956.912438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_timeout -r 1 [ 956.935991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_timeout -r 1 [ 956.962721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_timeout -r 1 [ 956.984783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 957.017841] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 957.020881] LNet: Skipped 5 previous similar messages [ 957.221179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.254416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 957.282677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.319598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.357408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.439838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.465994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.498925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.515645] LNet: 57996:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.523648] LNet: 57996:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 957.540324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.569192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 957.600527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 957.627995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 957.681914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 957.702490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 957.746155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 958.319417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_error -r 1 [ 958.351577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_error -r 1 [ 958.388867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_error -r 1 [ 958.426506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_error -r 1 [ 958.471165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_error -r 1 [ 958.501974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_error -r 1 [ 958.532881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_error -r 1 [ 958.561664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_error -r 1 [ 958.588618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 958.853356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.884520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 958.910757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.943672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.966510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.992377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 959.024464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 959.063826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 959.119141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 959.154453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 959.176108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 959.201294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 959.230700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 959.256941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 959.283729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 963.744217] LNet: 58192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 963.757517] LNetError: 58192:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 963.765891] LNet: Removed LNI 192.168.204.36@tcp [ 964.838984] LNet: Removed LNI 192.168.204.36@tcp1 [ 965.330207] Key type .llcrypt unregistered [ 965.331917] Key type ._llcrypt unregistered [ 977.648824] Key type ._llcrypt registered [ 977.650363] Key type .llcrypt registered [ 979.039277] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:24:24 (1766413464) [ 979.787391] Key type .llcrypt unregistered [ 979.792877] Key type ._llcrypt unregistered [ 984.352225] Key type ._llcrypt registered [ 984.353913] Key type .llcrypt registered [ 984.544711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.558973] alg: No test for adler32 (adler32-zlib) [ 985.493038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 985.536655] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 991.828598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 992.580437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_dropped -r 1 [ 992.622651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_dropped -r 1 [ 992.648994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 992.676286] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 993.007441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.051784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 993.093073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.144867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.190230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.242085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 993.293958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 993.338531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 993.370601] LNet: 59435:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 993.409222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 993.435247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 993.475543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 993.516138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 993.549902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 993.611478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 993.665308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 994.176651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_error -r 1 [ 994.216295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_error -r 1 [ 994.263278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 994.284362] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 994.532269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 994.593942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 994.644945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 994.690612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 994.728920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 994.774754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 994.817731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.873859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.901918] LNet: 59544:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 994.923764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.966546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 995.019394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 995.071237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 995.112760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 995.148517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 995.195263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 995.807153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_timeout -r 1 [ 995.855722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_timeout -r 1 [ 995.923938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 995.965634] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 996.346972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 996.393680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 996.444152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 996.504182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 996.563810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 996.590130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.636647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.708392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.735870] LNet: 59654:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.767923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 996.819202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 996.858930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 996.887450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 996.920348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 996.941889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 996.973963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 997.240348] LNet: 59694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.249616] LNetError: 59694:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.265962] LNet: Removed LNI 192.168.204.36@tcp [ 997.711881] Key type .llcrypt unregistered [ 997.715384] Key type ._llcrypt unregistered [ 1009.645268] Key type ._llcrypt registered [ 1009.646976] Key type .llcrypt registered [ 1010.688036] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:24:56 (1766413496) [ 1011.364575] Key type .llcrypt unregistered [ 1011.366268] Key type ._llcrypt unregistered [ 1014.918833] Key type ._llcrypt registered [ 1014.920329] Key type .llcrypt registered [ 1015.123143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.136746] alg: No test for adler32 (adler32-zlib) [ 1016.067600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1016.123935] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1022.343839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1029.613740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1029.665700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1029.716726] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1031.272755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_dropped -r 1 [ 1031.311889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e remote_dropped -r 1 [ 1031.353569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_dropped -r 1 [ 1031.378675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e remote_dropped -r 1 [ 1031.427427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e remote_dropped -r 1 [ 1031.470867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e remote_dropped -r 1 [ 1031.495932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e remote_dropped -r 1 [ 1031.539897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e remote_dropped -r 1 [ 1031.569740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 1031.582394] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1031.587806] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.136@tcp, 192.168.204.136@tcp1 [ 1031.842193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.881040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1031.927056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.968742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.995801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.018765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.049979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.085554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.112781] LNet: 61157:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.133701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.180405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.214437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.243371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.284404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.332982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.384439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1032.431653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.969921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_error -r 1 [ 1032.997985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e remote_error -r 1 [ 1033.038319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_error -r 1 [ 1033.085487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e remote_error -r 1 [ 1033.159490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e remote_error -r 1 [ 1033.216585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e remote_error -r 1 [ 1033.247237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e remote_error -r 1 [ 1033.299939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e remote_error -r 1 [ 1033.370840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 1033.394531] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1033.403332] LNet: Skipped 2 previous similar messages [ 1033.671897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.700689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.742244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.769418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.797476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.859810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.892221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.945366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.985617] LNet: 61281:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.015498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.050692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.099359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.139305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.188968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.249440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.295192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1034.339075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.758550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_timeout -r 1 [ 1034.793887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e remote_timeout -r 1 [ 1034.839899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_timeout -r 1 [ 1034.877191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e remote_timeout -r 1 [ 1034.920601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e remote_timeout -r 1 [ 1034.948886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e remote_timeout -r 1 [ 1034.975734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e remote_timeout -r 1 [ 1035.035917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e remote_timeout -r 1 [ 1035.092772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.136@tcp [ 1035.121622] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1035.375993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.411460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.481714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.522929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.550112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.583058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.620893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.668887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.683864] LNet: 61404:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.712841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.746510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.771905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.800548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.817816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.843325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.882912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.911459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1040.053743] LNet: 61481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.061911] LNetError: 61481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.119346] LNet: Removed LNI 192.168.204.36@tcp [ 1041.121384] LNetError: 61481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1041.140738] LNet: Removed LNI 192.168.204.36@tcp1 [ 1041.542957] Key type .llcrypt unregistered [ 1041.547189] Key type ._llcrypt unregistered [ 1056.089337] Key type ._llcrypt registered [ 1056.090641] Key type .llcrypt registered [ 1057.560392] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:25:42 (1766413542) [ 1058.030575] Key type .llcrypt unregistered [ 1058.037679] Key type ._llcrypt unregistered [ 1063.035295] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1063.212101] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1063.223250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1063.228341] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1065.658675] Key type ._llcrypt registered [ 1065.660344] Key type .llcrypt registered [ 1065.786365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1065.800207] alg: No test for adler32 (adler32-zlib) [ 1066.748149] LNet: 62554:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1066.760314] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1066.799044] LNet: 62568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.804820] LNetError: 62568:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.816752] LNet: Removed LNI 192.168.204.36@tcp [ 1067.445486] Key type .llcrypt unregistered [ 1067.447298] Key type ._llcrypt unregistered [ 1072.180983] Key type ._llcrypt registered [ 1072.182655] Key type .llcrypt registered [ 1072.361778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.372302] alg: No test for adler32 (adler32-zlib) [ 1073.323589] LNet: 62824:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1073.338617] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1073.340963] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1073.382575] LNet: 62838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.388630] LNetError: 62838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.393217] LNet: Removed LNI 192.168.204.36@tcp [ 1073.404748] LNet: Removed LNI 10.1.2.3@tcp1 [ 1073.951452] Key type .llcrypt unregistered [ 1073.953065] Key type ._llcrypt unregistered [ 1078.099823] Key type ._llcrypt registered [ 1078.101394] Key type .llcrypt registered [ 1078.263796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1078.279585] alg: No test for adler32 (adler32-zlib) [ 1079.175767] LNet: 63093:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1079.187715] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1079.191345] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1079.221387] LNet: 63107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.229641] LNetError: 63107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1079.234547] LNet: Removed LNI 192.168.204.36@tcp [ 1079.248585] LNet: Removed LNI 10.1.2.3@tcp [ 1079.865795] Key type .llcrypt unregistered [ 1079.869936] Key type ._llcrypt unregistered [ 1084.387966] Key type ._llcrypt registered [ 1084.389475] Key type .llcrypt registered [ 1084.538294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1084.552367] alg: No test for adler32 (adler32-zlib) [ 1085.489630] LNet: 63363:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1085.501873] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1085.504263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1085.532619] LNet: 63377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.541457] LNetError: 63377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.551161] LNet: Removed LNI 192.168.204.36@tcp [ 1085.563702] LNet: Removed LNI 10.1.2.3@tcp [ 1086.192213] Key type .llcrypt unregistered [ 1086.193889] Key type ._llcrypt unregistered [ 1091.078708] Key type ._llcrypt registered [ 1091.080190] Key type .llcrypt registered [ 1091.249027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.264678] alg: No test for adler32 (adler32-zlib) [ 1092.252688] LNet: 63637:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1092.267428] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1092.269648] LNetError: 63637:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1092.273156] LNetError: Error -100 starting up LNI tcp [ 1092.284918] LNet: Removed LNI 192.168.204.36@tcp [ 1092.887504] Key type .llcrypt unregistered [ 1092.889079] Key type ._llcrypt unregistered [ 1101.597699] Key type ._llcrypt registered [ 1101.601864] Key type .llcrypt registered [ 1102.715813] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:26:28 (1766413588) [ 1103.507535] Key type .llcrypt unregistered [ 1103.509319] Key type ._llcrypt unregistered [ 1107.714089] Key type ._llcrypt registered [ 1107.716206] Key type .llcrypt registered [ 1107.865184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1107.876855] alg: No test for adler32 (adler32-zlib) [ 1108.785132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1108.813635] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1115.081826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1115.759563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e network_timeout -r 1 [ 1115.781246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e network_timeout -r 1 [ 1115.805187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1115.823244] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1116.003720] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1116.033360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.064791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.104274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.145520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.178587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.220157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.272566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.316213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.341730] LNet: 64887:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1116.358915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.394368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.446268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.486331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.520236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.546019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.786428] LNet: 64925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1116.791284] LNetError: 64925:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1116.804843] LNet: Removed LNI 192.168.204.36@tcp [ 1117.196337] Key type .llcrypt unregistered [ 1117.198176] Key type ._llcrypt unregistered [ 1129.071571] Key type ._llcrypt registered [ 1129.072828] Key type .llcrypt registered [ 1129.290725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.305207] alg: No test for adler32 (adler32-zlib) [ 1130.185199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1130.209623] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1136.015575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1142.516476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.556182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1142.583748] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1143.932583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e network_timeout -r 1 [ 1143.967177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e network_timeout -r 1 [ 1144.011945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e network_timeout -r 1 [ 1144.039327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e network_timeout -r 1 [ 1144.084129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e network_timeout -r 1 [ 1144.117619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e network_timeout -r 1 [ 1144.144367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e network_timeout -r 1 [ 1144.172846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e network_timeout -r 1 [ 1144.257795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1144.280479] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1144.495872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1144.521986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1144.543201] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1144.545652] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.136@tcp [ 1144.556601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1144.589054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1144.614480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1144.638856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1144.666811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1144.702661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.727251] LNet: 65797:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1144.755020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.797445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.844387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.897594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.939800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.965769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1148.423858] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1148.731454] LNet: 65870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.743587] LNetError: 65870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.750398] LNet: Removed LNI 192.168.204.36@tcp [ 1148.761078] LNet: Removed LNI 192.168.204.36@tcp1 [ 1149.132675] Key type .llcrypt unregistered [ 1149.133971] Key type ._llcrypt unregistered [ 1161.391664] Key type ._llcrypt registered [ 1161.393241] Key type .llcrypt registered [ 1162.545647] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:27:28 (1766413648) [ 1162.873695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.886989] alg: No test for adler32 (adler32-zlib) [ 1163.763962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.808206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1163.830747] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1164.439860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1164.453965] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1165.209459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1165.272071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1165.310657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1165.330251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1165.387003] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1165.420768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1165.458808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.36@tcp [ 1165.495397] LNetError: 66967:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.36@tcp to recovery [ 1165.855565] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1165.855940] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1167.907516] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1168.972698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1171.999394] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1172.364969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1172.386897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.442720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1172.465893] LNet: 67074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.472941] LNetError: 67074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.478834] LNet: Removed LNI 192.168.204.36@tcp [ 1172.488498] LNet: Removed LNI 192.168.204.36@tcp1 [ 1172.506688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.553709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.590554] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1173.185738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1173.206104] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1173.928866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1173.955351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.007407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1174.055580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1174.088656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1174.115979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1174.140247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1174.164958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.36@tcp [ 1174.187097] LNetError: 67346:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.36@tcp to recovery [ 1174.560104] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1177.634939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1180.708938] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1180.714732] LNet: Skipped 1 previous similar message [ 1185.542827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1185.590193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.636943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.367708] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:27:57 (1766413677) [ 1191.466415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1191.482906] LNet: 68060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.494315] LNetError: 68060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.500127] LNetError: 68060:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1191.509171] LNet: Removed LNI 192.168.204.36@tcp [ 1191.531563] LNet: Removed LNI 192.168.204.36@tcp1 [ 1191.579920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.621658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1191.643757] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1192.293907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1192.336286] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1193.107455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1193.160081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1193.195960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1193.257163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1193.293671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1193.339673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1193.384799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.36@tcp --health 0 [ 1193.466603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1193.631123] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.36@tcp [ 1194.657033] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1194.666052] LNet: Skipped 1 previous similar message [ 1209.941741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1210.148224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1210.175840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.200644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.36@tcp --health 500 [ 1210.306561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1216.953245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1217.028723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1217.051912] LNet: 68563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.056471] LNetError: 68563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.065907] LNetError: 68563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1217.072294] LNet: Removed LNI 192.168.204.36@tcp [ 1217.087490] LNet: Removed LNI 192.168.204.36@tcp1 [ 1217.105290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.152880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1217.179842] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1217.794277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1217.807324] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1218.437389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1218.459130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1218.508968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1218.545275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1218.591615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1218.638148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1218.672099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1218.693347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.36@tcp --health 0 [ 1219.168712] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1219.175287] LNet: Skipped 3 previous similar messages [ 1220.024289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1230.279354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1230.310404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1230.337402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.154572] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:28:41 (1766413721) [ 1236.566227] LNet: 69608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.572128] LNetError: 69608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1236.576373] LNetError: 69608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1236.580451] LNet: Removed LNI 192.168.204.36@tcp [ 1236.592284] LNet: Removed LNI 192.168.204.36@tcp1 [ 1237.287828] Key type .llcrypt unregistered [ 1237.292296] Key type ._llcrypt unregistered [ 1241.810261] Key type ._llcrypt registered [ 1241.812280] Key type .llcrypt registered [ 1242.005757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1242.013699] alg: No test for adler32 (adler32-zlib) [ 1242.973127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1243.019529] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1250.106394] Lustre: DEBUG MARKER: Initial discovery [ 1250.227883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.136@tcp [ 1252.032598] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1252.178363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1252.233402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.136@tcp [ 1252.284789] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1252.306611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1252.340460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.136@tcp -r 1 -m GET -l 3 [ 1252.376942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.136@tcp -r 1 -m GET -e local_error [ 1252.421339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.136@tcp -r 1 -m PUT -l 6 [ 1253.280051] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1253.534465] Lustre: DEBUG MARKER: Force oleg436-client.virtnet to discover 192.168.204.136@tcp (in background) [ 1253.690388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.136@tcp [ 1254.831108] Lustre: DEBUG MARKER: Force oleg436-server to discover 192.168.204.36@tcp [ 1256.702626] Lustre: DEBUG MARKER: Wait for 70366 [ 1256.740720] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1257.925787] Lustre: DEBUG MARKER: Finished wait on 70366 [ 1257.992221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1258.052712] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1258.079264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1258.258358] LNet: 70815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.262814] LNetError: 70815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.286056] LNet: Removed LNI 192.168.204.36@tcp [ 1258.608891] Key type .llcrypt unregistered [ 1258.610867] Key type ._llcrypt unregistered [ 1269.124661] Key type ._llcrypt registered [ 1269.126848] Key type .llcrypt registered [ 1270.132489] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:29:16 (1766413756) [ 1270.486314] Key type .llcrypt unregistered [ 1270.488316] Key type ._llcrypt unregistered [ 1273.932179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1274.028569] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1274.031455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1274.034269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1276.413359] Key type ._llcrypt registered [ 1276.414986] Key type .llcrypt registered [ 1276.540989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1276.553255] alg: No test for adler32 (adler32-zlib) [ 1277.437888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1277.484983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1277.509457] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1278.029299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1278.041726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1285.121792] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:29:30 (1766413770) [ 1285.195539] LNet: 72874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.205317] LNetError: 72874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.208248] LNet: Removed LNI 192.168.204.36@tcp [ 1285.220578] LNet: Removed LNI 10.1.2.3@tcp [ 1285.657777] Key type .llcrypt unregistered [ 1285.659765] Key type ._llcrypt unregistered [ 1289.794374] Key type ._llcrypt registered [ 1289.795872] Key type .llcrypt registered [ 1289.959902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1289.968299] alg: No test for adler32 (adler32-zlib) [ 1290.856596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1290.902519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1290.926605] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1291.562433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1291.581747] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1292.390181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.36@tcp [ 1293.473770] LNetError: Unexpected error -22 connecting to 192.168.204.36@tcp at host 192.168.204.36:988 [ 1297.904952] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:29:43 (1766413783) [ 1298.103403] LNet: 74007:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.116509] LNetError: 74007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.167196] LNet: Removed LNI 192.168.204.36@tcp [ 1299.174281] LNetError: 74007:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1299.201767] LNet: Removed LNI 10.1.2.3@tcp [ 1299.787534] Key type .llcrypt unregistered [ 1299.789266] Key type ._llcrypt unregistered [ 1304.844608] Key type ._llcrypt registered [ 1304.846215] Key type .llcrypt registered [ 1305.141510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1305.158892] alg: No test for adler32 (adler32-zlib) [ 1306.128439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1306.235389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1306.274528] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1307.258581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1307.281578] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1308.461532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.36@tcp1 --nid 192.168.204.36@tcp2 [ 1312.826820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp1 192.168.204.36@tcp2 [ 1312.846499] LNetError: 74841:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.36@tcp1 to 192.168.204.36@tcp2 - Route table may be misconfigured [ 1312.876842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp2 192.168.204.36@tcp1 [ 1319.699464] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:30:05 (1766413805) [ 1319.887360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1319.931838] LNet: 75413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1319.949605] LNetError: 75413:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1319.967859] LNet: Removed LNI 192.168.204.36@tcp1 [ 1321.021493] LNet: Removed LNI 192.168.204.36@tcp2 [ 1321.108814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1321.182481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1321.207139] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1322.130148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1322.167762] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1323.121711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1323.172900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.36@tcp [ 1323.220865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp1 192.168.204.36@tcp1 [ 1323.261834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.36@tcp -d 192.168.204.36@tcp -e network_timeout [ 1323.297940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -e network_timeout [ 1323.341511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -e network_timeout [ 1323.375601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -e network_timeout [ 1323.417874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1323.460262] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1323.485916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1323.628718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1323.749514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1324.199926] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1330.697878] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:30:16 (1766413816) [ 1330.860572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1330.885907] LNet: 76274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1330.900652] LNetError: 76274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.903924] LNetError: 76274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1331.938580] LNet: Removed LNI 192.168.204.36@tcp [ 1331.947060] LNetError: 76274:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.967588] LNet: Removed LNI 192.168.204.36@tcp1 [ 1332.989142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.071664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1333.301745] LNet: 76318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.718817] Key type .llcrypt unregistered [ 1333.720320] Key type ._llcrypt unregistered [ 1342.585378] Key type ._llcrypt registered [ 1342.591441] Key type .llcrypt registered [ 1343.885109] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:30:29 (1766413829) [ 1344.258497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.289252] alg: No test for adler32 (adler32-zlib) [ 1345.193453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.622410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1348.416825] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:30:34 (1766413834) [ 1348.550317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1348.586461] LNet: 77408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.620944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.745900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1348.768419] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1349.563207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1349.626126] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1350.838110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1350.882903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp1 [ 1350.933862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp1 [ 1357.246696] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:30:42 (1766413842) [ 1359.267350] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg436-server [ 1361.170103] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:30:46 (1766413846) [ 1362.851602] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg436-server [ 1364.546909] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:30:50 (1766413850) [ 1366.322230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg436-server [ 1368.043052] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:30:53 (1766413853) [ 1369.942864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg436-server [ 1371.810734] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:30:57 (1766413857) [ 1373.475903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg436-server [ 1374.969149] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:31:00 (1766413860) [ 1376.591228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg436-server [ 1378.218538] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:31:03 (1766413863) [ 1379.866829] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg436-server [ 1381.303438] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:31:07 (1766413867) [ 1382.951720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg436-server [ 1384.707955] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:31:10 (1766413870) [ 1386.527539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg436-server [ 1388.176670] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:31:13 (1766413873) [ 1388.273706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.317990] LNet: 81284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.333720] LNetError: 81284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.345211] LNet: Removed LNI 192.168.204.36@tcp [ 1389.418174] LNet: Removed LNI 192.168.204.36@tcp1 [ 1389.452465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.528517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1389.577807] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1390.433751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1390.519229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1390.733210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.762075] LNet: 81438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1390.772584] LNetError: 81438:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.775941] LNetError: 81438:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1390.789816] LNet: Removed LNI 192.168.204.36@tcp [ 1390.807286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.885774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1390.934371] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1391.768479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1392.012435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.034188] LNetError: 81583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.046737] LNet: Removed LNI 192.168.204.36@tcp [ 1392.066476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.121425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.154318] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1392.883787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1392.969764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1393.250769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.301807] LNet: 81730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.304301] LNet: 81730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1394.352490] LNet: Removed LNI 192.168.204.36@tcp [ 1394.377901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.479261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1394.529265] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1395.534760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1395.874426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.900428] LNetError: 81876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.905786] LNetError: 81876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1395.916514] LNet: Removed LNI 192.168.204.36@tcp [ 1395.942911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1395.999117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.021975] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1396.871613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1396.941088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1397.266940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1397.324403] LNet: 82023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.334831] LNet: 82023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1398.382789] LNet: Removed LNI 192.168.204.36@tcp [ 1398.442788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.525258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1398.569787] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1399.550469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1401.040818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.084136] LNetError: 82178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.091188] LNetError: 82178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1401.113666] LNet: Removed LNI 192.168.204.36@tcp [ 1401.162718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.243586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.288511] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1402.060907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1402.179747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1402.486209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1403.563796] LNet: Removed LNI 192.168.204.36@tcp [ 1403.605891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.680800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1403.703875] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1404.694562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1407.376065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1407.418134] LNet: 82489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.420883] LNet: 82489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1408.491244] LNet: Removed LNI 192.168.204.36@tcp [ 1408.546880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.633324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1408.664757] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1409.563517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1409.637085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1409.979560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.010096] LNetError: 82636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.013532] LNetError: 82636:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1410.034856] LNet: Removed LNI 192.168.204.36@tcp [ 1410.060990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.124728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1410.149524] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1410.941082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1412.332345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.414176] LNet: Removed LNI 192.168.204.36@tcp [ 1413.444063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.524235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.550748] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1414.465940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1414.546625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1414.863643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.914623] LNet: Removed LNI 192.168.204.36@tcp [ 1415.955195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1416.051841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1416.085666] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1416.905702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1417.258965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1418.344619] LNet: Removed LNI 192.168.204.36@tcp [ 1418.380712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.446545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1418.484364] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1419.219561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1419.288445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.321577] LNet: Removed LNI 192.168.204.36@tcp [ 1419.344599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1419.414604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1419.450943] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1420.321707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1427.147662] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:31:52 (1766413912) [ 1427.496451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.515334] LNet: 83922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1427.518731] LNet: 83922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1427.528073] LNetError: 83922:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.530478] LNetError: 83922:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1427.541163] LNet: Removed LNI 192.168.204.36@tcp [ 1427.555910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.611711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1427.636452] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/180] [ 1428.898701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.932320] LNet: Removed LNI 192.168.204.36@tcp231 [ 1428.948556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.042436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1429.084937] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/180] [ 1429.093986] LNet: 84068:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1429.483578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1429.551779] LNet: Removed LNI 192.168.204.36@tcp231 [ 1429.569384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1429.590433] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/180] [ 1430.820790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.881100] LNet: Removed LNI 192.168.204.36@tcp231 [ 1430.897350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.999435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1431.047297] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/0] [ 1431.053343] LNet: 84220:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1431.406978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1431.436623] LNet: Removed LNI 192.168.204.36@tcp231 [ 1431.460653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1431.481192] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/0] [ 1432.438875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1432.469483] LNet: Removed LNI 192.168.204.36@tcp231 [ 1432.493316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.572454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1432.595554] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/60] [ 1432.604307] LNet: 84372:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.967812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1432.998669] LNet: Removed LNI 192.168.204.36@tcp231 [ 1433.019421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1433.042261] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/60] [ 1434.033149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.070878] LNet: Removed LNI 192.168.204.36@tcp231 [ 1434.092615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.172589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1434.200465] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/180] [ 1434.509184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1434.544118] LNet: Removed LNI 192.168.204.36@tcp231 [ 1434.568421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1434.638576] LNet: Added LNI 192.168.204.36@tcp231 [8/256/0/180] [ 1441.952375] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:32:07 (1766413927) [ 1442.086762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.124226] LNet: Removed LNI 192.168.204.36@tcp231 [ 1442.168318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.260972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1442.289547] LNet: Added LNI 192.168.204.36@tcp232 [8/256/0/180] [ 1443.138145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1443.173856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1443.200128] LNet: Removed LNI 192.168.204.36@tcp232 [ 1443.213961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1443.242572] LNet: Added LNI 192.168.204.36@tcp232 [8/256/0/180] [ 1443.925899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1443.953597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1443.983718] LNet: Removed LNI 192.168.204.36@tcp232 [ 1444.004862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1444.041760] LNet: Added LNI 192.168.204.36@tcp232 [8/256/0/180] [ 1444.776658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1444.810295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1444.852649] LNet: Removed LNI 192.168.204.36@tcp232 [ 1444.874389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1444.894804] LNet: Added LNI 192.168.204.36@tcp232 [8/256/0/180] [ 1445.625698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1445.660495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1445.692313] LNet: Removed LNI 192.168.204.36@tcp232 [ 1445.716700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1451.799859] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:32:17 (1766413937) [ 1453.284611] Key type .llcrypt unregistered [ 1453.287783] Key type ._llcrypt unregistered [ 1459.345771] Key type ._llcrypt registered [ 1459.347803] Key type .llcrypt registered [ 1459.546931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.561929] alg: No test for adler32 (adler32-zlib) [ 1460.494308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1460.529749] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1466.757437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1474.507444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.589513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1474.628338] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1476.296850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_interrupt -r 2 [ 1476.321362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_interrupt -r 2 [ 1476.361705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_interrupt -r 2 [ 1476.407923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_interrupt -r 2 [ 1476.462336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_interrupt -r 2 [ 1476.503923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_interrupt -r 2 [ 1476.541882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_interrupt -r 2 [ 1476.573378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_interrupt -r 2 [ 1476.615180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.136@tcp [ 1476.639341] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1476.647127] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1476.920983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1476.956954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1476.986774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1477.042312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1477.084563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1477.122755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1477.154521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1477.203863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1477.256885] LNet: 87018:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1477.276666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1477.311516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1477.348292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1477.393045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1477.446880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1477.511363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1478.082264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_dropped -r 2 [ 1478.129729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_dropped -r 2 [ 1478.198632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_dropped -r 2 [ 1478.250891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_dropped -r 2 [ 1478.319876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_dropped -r 2 [ 1478.361114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_dropped -r 2 [ 1478.427208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_dropped -r 2 [ 1478.458544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_dropped -r 2 [ 1478.506590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.136@tcp [ 1478.535730] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1478.910858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1478.952095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1478.996202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1479.047494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1479.095274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1479.152822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1479.196866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1479.251104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1479.296677] LNet: 87139:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1479.347945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1479.398949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1479.482308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1479.536392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1479.605680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1479.637802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1480.301391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_aborted -r 2 [ 1480.366672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_aborted -r 2 [ 1480.428444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_aborted -r 2 [ 1480.510295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_aborted -r 2 [ 1480.563108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_aborted -r 2 [ 1480.634413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_aborted -r 2 [ 1480.690807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_aborted -r 2 [ 1480.762058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_aborted -r 2 [ 1480.820688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.136@tcp [ 1480.868343] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1481.213650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1481.271527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1481.325045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1481.366109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1481.402303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1481.443357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1481.523105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1481.570285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1481.604129] LNet: 87259:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1481.621166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1481.703961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1481.735489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.795283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1481.859260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1481.903874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1482.414856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_no_route -r 2 [ 1482.453888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_no_route -r 2 [ 1482.486877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_no_route -r 2 [ 1482.539790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_no_route -r 2 [ 1482.597620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_no_route -r 2 [ 1482.654272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_no_route -r 2 [ 1482.717742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_no_route -r 2 [ 1482.813766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_no_route -r 2 [ 1482.910664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.136@tcp [ 1482.938445] LNet: There was an unexpected network error while writing to 192.168.204.136: rc = -22 [ 1483.234658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1483.266755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1483.294803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1483.340576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1483.365308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1483.384742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1483.437215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1483.484887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1483.548680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1483.589819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1483.649409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.701370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.752106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1483.809156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1484.418988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e local_timeout -r 2 [ 1484.463693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e local_timeout -r 2 [ 1484.511073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e local_timeout -r 2 [ 1484.585663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e local_timeout -r 2 [ 1484.640896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e local_timeout -r 2 [ 1484.730288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e local_timeout -r 2 [ 1484.832605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e local_timeout -r 2 [ 1484.920196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e local_timeout -r 2 [ 1484.991590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.136@tcp [ 1485.287587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1485.331590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1485.372283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1485.412038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1485.437142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1485.470528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1485.504267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1485.584349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1485.597856] LNet: 87500:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1485.602161] LNet: 87500:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1485.652782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1485.685391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1485.712847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1485.732907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1485.769721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1485.822878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1486.356434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp -m GET -e remote_dropped -r 2 [ 1486.414697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.136@tcp1 -m GET -e remote_dropped -r 2 [ 1486.475713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp -m GET -e remote_dropped -r 2 [ 1486.546167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp -d 192.168.204.36@tcp1 -m GET -e remote_dropped -r 2 [ 1486.606744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp -m GET -e remote_dropped -r 2 [ 1486.670555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.36@tcp1 -d 192.168.204.136@tcp1 -m GET -e remote_dropped -r 2 [ 1486.738468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp -m GET -e remote_dropped -r 2 [ 1486.798873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.36@tcp1 -d 192.168.204.36@tcp1 -m GET -e remote_dropped -r 2 [ 1486.888481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.36@tcp 192.168.204.136@tcp [ 1486.936589] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.136@tcp [ 1487.225393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1487.296801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1487.354905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1487.445637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1487.519823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1487.570629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1487.621651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1487.684200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1487.727987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1487.791461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1487.852967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1487.913960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1487.974480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1488.059638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1493.974422] LNet: 87693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.984560] LNetError: 87693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.993146] LNet: Removed LNI 192.168.204.36@tcp [ 1494.005969] LNet: Removed LNI 192.168.204.36@tcp1 [ 1494.414644] Key type .llcrypt unregistered [ 1494.416447] Key type ._llcrypt unregistered [ 1509.976816] Key type ._llcrypt registered [ 1509.981294] Key type .llcrypt registered [ 1511.077328] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 09:33:16 (1766413996) [ 1511.618702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1511.644538] alg: No test for adler32 (adler32-zlib) [ 1512.661938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.751805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1512.791841] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1513.646873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1513.662972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.36@tcp [ 1513.725693] LNetError: 88668:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.36@tcp to recovery [ 1513.819474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.36@tcp [ 1513.920810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1519.719925] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 09:33:25 (1766414005) [ 1519.876320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.910451] LNet: 89257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.919554] LNetError: 89257:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.939140] LNet: Removed LNI 192.168.204.36@tcp [ 1519.966539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.024334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1520.056397] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1520.856615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1520.885897] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1521.719757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.36@tcp [ 1521.759900] LNetError: 89523:0:(api-ni.c:4348:lnet_ni_set_healthv()) manually adding local NI 192.168.204.36@tcp to recovery [ 1522.079391] LNet: 1 local NIs in recovery (showing 1): 192.168.204.36@tcp [ 1522.082033] LNet: There was an unexpected network error while writing to 192.168.204.36: rc = -22 [ 1522.991615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.36@tcp [ 1530.500079] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 09:33:36 (1766414016) [ 1532.185486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg436-server [ 1533.981284] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 09:33:39 (1766414019) [ 1534.064682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.100033] LNet: 90464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1534.112973] LNetError: 90464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.137459] LNet: Removed LNI 192.168.204.36@tcp [ 1535.147879] LNetError: 90464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1535.175486] LNet: Removed LNI 192.168.204.36@tcp1 [ 1535.213352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.290261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1535.349381] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1536.207874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1536.291449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1536.335271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1536.403437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1536.455047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1536.494736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1541.932665] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 09:33:47 (1766414027) [ 1542.049658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.068520] LNet: 91191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.076329] LNetError: 91191:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.101046] LNet: Removed LNI 192.168.204.36@tcp [ 1542.137817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.225958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.269148] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1548.567750] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 09:33:54 (1766414034) [ 1548.640097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.676679] LNet: 91891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.694963] LNetError: 91891:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.716281] LNet: Removed LNI 192.168.204.36@tcp [ 1548.752648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.805690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1548.832488] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1554.970450] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 09:34:00 (1766414040) [ 1556.310563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1557.872217] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 09:34:03 (1766414043) [ 1558.413954] LNet: 92933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1558.419193] LNetError: 92933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.434506] LNet: Removed LNI 192.168.204.36@tcp [ 1559.028763] Key type .llcrypt unregistered [ 1559.031962] Key type ._llcrypt unregistered [ 1563.963282] Key type ._llcrypt registered [ 1563.965472] Key type .llcrypt registered [ 1564.192096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1564.208975] alg: No test for adler32 (adler32-zlib) [ 1565.143455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1565.165466] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1572.383627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1576.772984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.136@tcp [ 1577.142806] LNet: 93344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1577.154139] LNetError: 93344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1577.168301] LNet: Removed LNI 192.168.204.36@tcp [ 1577.598444] Key type .llcrypt unregistered [ 1577.600072] Key type ._llcrypt unregistered [ 1587.394545] Key type ._llcrypt registered [ 1587.401138] Key type .llcrypt registered [ 1588.997463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1590.690539] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1592.690304] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 09:34:38 (1766414078) [ 1592.991171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1593.005792] alg: No test for adler32 (adler32-zlib) [ 1593.976611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.081414] LNet: 94427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.745672] Key type .llcrypt unregistered [ 1594.747313] Key type ._llcrypt unregistered [ 1599.580040] Key type ._llcrypt registered [ 1599.592208] Key type .llcrypt registered [ 1599.756211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1599.770190] alg: No test for adler32 (adler32-zlib) [ 1600.699590] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1600.701860] LNet: 94680:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1600.708939] LNet: 94680:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1600.760219] LNetError: 94692:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.37@tcp is being used as a gateway but routing feature is not turned on [ 1600.779866] LNetError: 94692:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.37@tcp has gone from up to down [ 1601.237980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1601.249987] LNet: 94739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.263186] LNetError: 94739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.280165] LNet: Removed LNI 192.168.204.36@tcp [ 1601.302582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1601.354589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1601.379837] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1601.387418] LNet: 94746:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1601.392617] LNet: 94746:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1601.397965] LNetError: 94746:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.38@tcp is a gateway. Can not delete it [ 1601.413540] LNetError: 94746:0:(peer.c:2217:lnet_del_peer_ni()) 192.168.204.38@tcp is a router. Can not be deleted [ 1601.424715] LNet: 94746:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1601.439623] LNetError: 94743:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.40@tcp is being used as a gateway but routing feature is not turned on [ 1601.451554] LNetError: 94743:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 1601.460839] LNetError: 94743:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.40@tcp has gone from up to down [ 1601.472690] LNetError: 94743:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1607.746639] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 09:34:53 (1766414093) [ 1609.640943] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg436-server [ 1611.542410] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 09:34:57 (1766414097) [ 1613.306830] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg436-server [ 1615.465837] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 09:35:00 (1766414100) [ 1617.404361] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 1619.481052] Key type lgssc registered [ 1620.895878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1640.621463] Key type lgssc unregistered [ 1641.037346] LNet: 97162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1641.044798] LNetError: 97162:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1641.064127] LNet: Removed LNI 192.168.204.36@tcp [ 1641.555673] Key type .llcrypt unregistered [ 1641.558348] Key type ._llcrypt unregistered [ 1653.701802] Key type ._llcrypt registered [ 1653.706043] Key type .llcrypt registered [ 1654.906155] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 09:35:40 (1766414140) [ 1655.183477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1655.197170] alg: No test for adler32 (adler32-zlib) [ 1656.097105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.151398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1656.177593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1656.221851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1656.253373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1661.690879] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 09:35:47 (1766414147) [ 1661.767748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.813652] LNet: 98561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.865659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.952953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.975894] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1662.753054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.770400] LNet: 98710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.782149] LNetError: 98710:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.801446] LNet: Removed LNI 192.168.204.36@tcp [ 1662.824106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.895552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1662.945731] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1663.374344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.394191] LNetError: 98738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1663.417466] LNet: Removed LNI 192.168.204.36@tcp [ 1663.433477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.478532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1663.497893] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1669.433458] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 09:35:55 (1766414155) [ 1669.611719] LNet: 99325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.615922] LNet: 99325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1669.620335] LNetError: 99325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1669.649182] LNet: Removed LNI 192.168.204.36@tcp [ 1670.321707] Key type .llcrypt unregistered [ 1670.323438] Key type ._llcrypt unregistered [ 1675.104844] Key type ._llcrypt registered [ 1675.106831] Key type .llcrypt registered [ 1675.300765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1675.319412] alg: No test for adler32 (adler32-zlib) [ 1676.247501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.337585] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1676.339414] LNet: *** cfs_fail_loc=e003, val=100*** [ 1677.068452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1677.087508] LNet: *** cfs_fail_loc=e003, val=100*** [ 1677.089368] LNet: Skipped 2 previous similar messages [ 1677.098724] Ping reply received with mapping: [ 1677.100402] found metadata reply NI[192.168.204.36@tcp] [ 1677.102274] 24 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1677.106296] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1677.109421] 5a 5a 5a 5a 5a 5a 5a 5a [ 1677.223053] LNet: 99722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.226796] LNetError: 99722:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.249690] LNet: Removed LNI 192.168.204.36@tcp [ 1678.941087] Key type .llcrypt unregistered [ 1678.945595] Key type ._llcrypt unregistered [ 1688.580085] Key type ._llcrypt registered [ 1688.581775] Key type .llcrypt registered [ 1690.045505] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 09:36:15 (1766414175) [ 1690.391284] Key type .llcrypt unregistered [ 1690.393070] Key type ._llcrypt unregistered [ 1696.046856] Key type ._llcrypt registered [ 1696.058874] Key type .llcrypt registered [ 1696.444200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1696.473758] alg: No test for adler32 (adler32-zlib) [ 1697.547910] LNetError: 100758:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1698.085765] Key type .llcrypt unregistered [ 1698.088096] Key type ._llcrypt unregistered [ 1703.686355] Key type ._llcrypt registered [ 1703.687785] Key type .llcrypt registered [ 1704.866929] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 09:36:30 (1766414190) [ 1705.384270] Key type .llcrypt unregistered [ 1705.388092] Key type ._llcrypt unregistered [ 1710.889261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1710.891406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1713.443230] Key type ._llcrypt registered [ 1713.444802] Key type .llcrypt registered [ 1713.682076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1713.698209] alg: No test for adler32 (adler32-zlib) [ 1714.614766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.686047] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1723.440162] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 09:36:49 (1766414209) [ 1723.500684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.519741] LNet: 102524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.529048] LNetError: 102524:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1723.564352] LNet: Removed LNI 10.1.2.3@tcp [ 1723.589946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1724.174372] Key type .llcrypt unregistered [ 1724.175784] Key type ._llcrypt unregistered [ 1729.710096] Key type ._llcrypt registered [ 1729.711754] Key type .llcrypt registered [ 1729.969796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1729.996651] alg: No test for adler32 (adler32-zlib) [ 1730.903754] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1730.951690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1731.073065] LNet: 102814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1731.079193] LNetError: 102814:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1731.089558] LNet: Removed LNI 192.168.204.36@tcp [ 1731.623077] Key type .llcrypt unregistered [ 1731.624576] Key type ._llcrypt unregistered [ 1740.678579] Key type ._llcrypt registered [ 1740.680811] Key type .llcrypt registered [ 1741.856699] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 09:37:07 (1766414227) [ 1742.368311] Key type .llcrypt unregistered [ 1742.370132] Key type ._llcrypt unregistered [ 1747.087629] Key type ._llcrypt registered [ 1747.088823] Key type .llcrypt registered [ 1747.261325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.280958] alg: No test for adler32 (adler32-zlib) [ 1753.842043] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 09:37:19 (1766414239) [ 1753.956500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.001460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.082970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.117725] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1754.927612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.956613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1754.982525] LNet: Removed LNI 192.168.204.36@tcp [ 1755.241673] LNet: 104549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.571342] Key type .llcrypt unregistered [ 1755.573047] Key type ._llcrypt unregistered [ 1765.090503] Key type ._llcrypt registered [ 1765.091619] Key type .llcrypt registered [ 1766.368763] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 09:37:32 (1766414252) [ 1766.676343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1766.686603] alg: No test for adler32 (adler32-zlib) [ 1767.644028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.721734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.743689] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1768.400501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1768.436513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.36@tcp [ 1768.455954] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1768.458668] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1768.461596] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1768, 0, 0 [ 1768.466185] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1768.468921] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1768.472743] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1768.476342] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1768, 0, 0 [ 1768.480145] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1768.483090] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1768.485398] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1768.488292] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1768, 0, 1818 [ 1768.491904] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1768.495804] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1768.499452] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1768.503479] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1768, 0, 1818 [ 1768.507705] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1774.395640] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 09:37:40 (1766414260) [ 1774.997432] LNet: 106086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.013653] LNetError: 106086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.037934] LNet: Removed LNI 192.168.204.36@tcp [ 1776.559766] Key type .llcrypt unregistered [ 1776.561830] Key type ._llcrypt unregistered [ 1781.075388] Key type ._llcrypt registered [ 1781.076919] Key type .llcrypt registered [ 1781.313661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1781.343576] alg: No test for adler32 (adler32-zlib) [ 1782.288541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1782.351684] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1788.261183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1794.619194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.665317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1794.698538] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 1795.625560] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1795.808546] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1795.811923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1795.813708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1798.084746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1798.105683] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1807.335114] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 09:38:13 (1766414293) [ 1807.462442] LNet: 107584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.468341] LNetError: 107584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.474407] LNet: Removed LNI 192.168.204.36@tcp [ 1808.543171] LNet: Removed LNI 192.168.204.36@tcp1 [ 1808.545206] LNetError: 107584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1808.563125] LNetError: 107584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1808.574522] LNet: Removed LNI 10.1.2.3@tcp99 [ 1809.112320] Key type .llcrypt unregistered [ 1809.115850] Key type ._llcrypt unregistered [ 1814.632854] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1814.645128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1817.080120] Key type ._llcrypt registered [ 1817.081668] Key type .llcrypt registered [ 1817.244618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1817.261405] alg: No test for adler32 (adler32-zlib) [ 1818.155312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.217859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1818.244485] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1818.934667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1818.972077] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1821.095230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1821.134457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1821.174863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1821.203965] LNetError: 108331:0:(peer.c:2225:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1821.255111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1821.294943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1821.666275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1821.727622] LNet: 108351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.735347] LNetError: 108351:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.751372] LNet: Removed LNI 192.168.204.36@tcp [ 1822.755274] LNetError: 108351:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1822.766578] LNet: Removed LNI 10.1.2.3@tcp [ 1823.342953] Key type .llcrypt unregistered [ 1823.350868] Key type ._llcrypt unregistered [ 1832.594183] Key type ._llcrypt registered [ 1832.596698] Key type .llcrypt registered [ 1833.646340] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 09:38:39 (1766414319) [ 1834.048773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1834.059260] alg: No test for adler32 (adler32-zlib) [ 1834.983889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.028220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.048848] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1835.651273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.36@tcp [ 1835.741237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg436-client@tcp [ 1841.906954] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 09:38:47 (1766414327) [ 1842.022392] LNet: 109872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.044454] LNetError: 109872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.081942] LNet: Removed LNI 192.168.204.36@tcp [ 1842.728535] Key type .llcrypt unregistered [ 1842.732217] Key type ._llcrypt unregistered [ 1847.668913] Key type ._llcrypt registered [ 1847.670306] Key type .llcrypt registered [ 1847.799821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.813128] alg: No test for adler32 (adler32-zlib) [ 1848.666476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1848.689478] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1848.713787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1854.119980] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 09:38:59 (1766414339) [ 1855.651144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1857.132657] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 09:39:03 (1766414343) [ 1858.428845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1859.826052] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 09:39:05 (1766414345) [ 1859.971122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.003361] LNet: 111292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.018358] LNetError: 111292:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.068304] LNet: Removed LNI 192.168.204.36@tcp [ 1860.090501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.140156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.168274] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1860.830054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1860.875232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1866.710728] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 09:39:12 (1766414352) [ 1866.881415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.901689] LNet: 112003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.908248] LNetError: 112003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1867.947566] LNet: Removed LNI 192.168.204.36@tcp [ 1867.974898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.052511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1868.086659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1868.324414] LNet: 112040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.661506] Key type .llcrypt unregistered [ 1868.663179] Key type ._llcrypt unregistered [ 1878.255799] Key type ._llcrypt registered [ 1878.257344] Key type .llcrypt registered [ 1879.482275] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 09:39:25 (1766414365) [ 1879.905608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.920311] alg: No test for adler32 (adler32-zlib) [ 1880.892339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1880.972731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1881.021886] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1881.744559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1881.806120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1887.522771] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 09:39:33 (1766414373) [ 1887.624247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1887.643262] LNet: 113551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.648485] LNetError: 113551:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.666461] LNet: Removed LNI 192.168.204.36@tcp [ 1887.700592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.736144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1887.765817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1893.272321] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 09:39:39 (1766414379) [ 1893.385670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.417739] LNet: 114131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.438956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.517394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1893.777552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1893.807890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1893.880661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1894.248368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1894.280562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1894.341363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.378820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.426240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1894.659820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1894.683135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1894.708691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1894.967321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1894.985392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1901.085874] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 09:39:46 (1766414386) [ 1901.248862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.285368] LNet: 114769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.288915] LNet: 114769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1901.326064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.424812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1901.490968] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1902.544721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1902.593679] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1903.541118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1903.586787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1903.784928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1903.802868] LNet: 115049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.811975] LNetError: 115049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.815898] LNet: Removed LNI 192.168.204.36@tcp [ 1903.841240] LNet: Removed LNI 192.168.204.36@tcp2 [ 1903.868909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1903.928517] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1903.933675] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1903.936182] LNet: 115055:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1904.479052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1904.670323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.696773] LNet: Removed LNI 192.168.204.36@tcp [ 1905.698542] LNetError: 115092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.710208] LNetError: 115092:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1905.731979] LNet: Removed LNI 192.168.204.36@tcp2 [ 1905.756623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.821381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.856037] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1906.473661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1906.493076] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1907.220931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1907.257170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1907.307913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1907.485259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.511813] LNet: Removed LNI 192.168.204.36@tcp [ 1908.521207] LNetError: 115377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1908.531667] LNetError: 115377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1908.544913] LNet: Removed LNI 192.168.204.36@tcp2 [ 1908.569224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.633550] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1908.639600] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1908.646319] LNet: 115383:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1909.266126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1909.484302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.513616] LNet: 115420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.520118] LNet: 115420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1909.525860] LNet: Removed LNI 192.168.204.36@tcp [ 1910.577783] LNet: Removed LNI 192.168.204.36@tcp2 [ 1910.608977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1910.656677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1910.681652] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1911.349354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1911.378821] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1912.285455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1912.334148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1912.532347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.549289] LNetError: 115700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1912.559288] LNetError: 115700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1912.567156] LNet: Removed LNI 192.168.204.36@tcp [ 1912.598690] LNet: Removed LNI 192.168.204.36@tcp2 [ 1912.616687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1912.669764] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1912.678455] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1912.682648] LNet: 115706:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1913.301310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1913.532204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1914.591168] LNet: Removed LNI 192.168.204.36@tcp [ 1914.608524] LNet: Removed LNI 192.168.204.36@tcp2 [ 1914.629780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1914.681815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1914.709956] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1915.386201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1915.398124] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1916.323275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1916.366542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1916.455439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1916.678342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.727170] LNet: Removed LNI 192.168.204.36@tcp [ 1918.757537] LNet: Removed LNI 192.168.204.36@tcp2 [ 1918.779494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.850274] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1918.854359] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1918.858645] LNet: 116034:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1919.401658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1919.594775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.624948] LNet: 116071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.629960] LNet: 116071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1919.642500] LNet: Removed LNI 192.168.204.36@tcp [ 1920.681071] LNet: Removed LNI 192.168.204.36@tcp2 [ 1920.706817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.802855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.864836] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1921.742104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1921.759208] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1922.655081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1922.706528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1922.759746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1923.147971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.166825] LNetError: 116368:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1923.173135] LNetError: 116368:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1923.182267] LNet: Removed LNI 192.168.204.36@tcp [ 1923.198809] LNet: Removed LNI 192.168.204.36@tcp2 [ 1923.217110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1923.272918] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1923.278414] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1923.283778] LNet: 116374:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1923.802547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1924.119242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.151164] LNet: Removed LNI 192.168.204.36@tcp [ 1925.165685] LNet: Removed LNI 192.168.204.36@tcp2 [ 1925.182522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.276516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1925.306848] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1926.166275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1926.189947] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1927.015683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1927.053281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1927.082493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1927.135034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1927.526221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.607328] LNet: Removed LNI 192.168.204.36@tcp [ 1929.645143] LNet: Removed LNI 192.168.204.36@tcp2 [ 1929.670074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.737171] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1929.742438] LNet: Added LNI 192.168.204.36@tcp2 [8/256/0/180] [ 1930.043522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1935.791818] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1937.129818] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 09:40:23 (1766414423) [ 1937.285863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.326763] LNet: 117492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.337941] LNet: 117492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1937.347197] LNet: Removed LNI 192.168.204.36@tcp [ 1938.406684] LNet: Removed LNI 192.168.204.36@tcp2 [ 1938.427574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.617613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.641871] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1939.215870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1939.233544] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1939.907552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1939.927425] LNetError: 117766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1939.931617] LNetError: 117766:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1939.935826] LNet: Removed LNI 192.168.204.36@tcp [ 1939.950049] LNet: Removed LNI 10.1.2.3@tcp [ 1939.966889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1940.028802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1940.055420] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1940.061629] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1940.323531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1940.336857] LNet: Removed LNI 192.168.204.36@tcp [ 1940.345386] LNet: Removed LNI 10.1.2.3@tcp [ 1945.174866] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 09:40:31 (1766414431) [ 1945.240130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1945.283858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.386267] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1945.523632] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1945.525633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1945.527946] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1947.784599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.813213] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1948.355804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1948.370459] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1949.860392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.36@tcp [ 1949.901878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1949.947501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.36@tcp -m PUT -e local_timeout -r 1 [ 1949.991439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1950.031238] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1950.034488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1950.231238] LNetError: 118431:0:(lib-move.c:4947:lnet_parse()) 192.168.204.36@tcp, src 192.168.204.36@tcp: Dropping PUT (error -108 looking up sender) [ 1952.287490] LNet: Removed LNI 192.168.204.36@tcp [ 1952.308212] LNet: Removed LNI 10.1.2.3@tcp [ 1952.714768] Key type .llcrypt unregistered [ 1952.717817] Key type ._llcrypt unregistered [ 1960.372423] Key type ._llcrypt registered [ 1960.373778] Key type .llcrypt registered [ 1961.334431] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 09:40:47 (1766414447) [ 1961.574279] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1961.584118] alg: No test for adler32 (adler32-zlib) [ 1962.462863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.766579] LNet: 119753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.262679] Key type .llcrypt unregistered [ 1963.264128] Key type ._llcrypt unregistered [ 1967.260103] Key type ._llcrypt registered [ 1967.261698] Key type .llcrypt registered [ 1967.387949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1967.409231] alg: No test for adler32 (adler32-zlib) [ 1968.303017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1968.322884] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 1973.775449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 1973.884613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1974.265121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.136@tcp [ 1974.410389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1974.435530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1974.458264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1974.482540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1974.495120] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.136@tcp [ 1974.509105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1974.528318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1974.559432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1974.600288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1974.615470] LNet: 120199:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1974.631381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1974.669627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1974.689933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1974.719095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1974.757855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1974.797391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1975.222525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1975.402151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1975.424080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1975.450395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1975.480865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1975.504177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1975.527354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1975.564560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1975.589697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1975.602411] LNet: 120316:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1975.618386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1975.635759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1975.664963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1975.687793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1975.737668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1975.757850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1976.238523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.136@tcp [ 1976.394813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1976.420696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1976.444350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1976.471920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1976.497069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1976.522645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1976.543588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1976.567991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1976.590065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1976.608974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1976.631560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1976.654840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1976.685925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1976.715877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1977.286263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1977.503405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1977.538145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1977.565899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1977.639787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1977.681879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1977.704280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1977.731318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1977.765295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1977.782889] LNet: 120550:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1977.795027] LNet: 120550:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1977.808688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1977.839393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1977.859956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1977.887507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1977.918242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1977.951676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1978.176525] LNet: 120595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1978.184838] LNetError: 120595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.202606] LNet: Removed LNI 192.168.204.36@tcp [ 1978.446667] Key type .llcrypt unregistered [ 1978.449148] Key type ._llcrypt unregistered [ 1988.774076] Key type ._llcrypt registered [ 1988.775626] Key type .llcrypt registered [ 1989.813463] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 09:41:15 (1766414475) [ 1990.130247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1990.146752] alg: No test for adler32 (adler32-zlib) [ 1991.073167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1991.401667] LNet: 121480:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.860735] Key type .llcrypt unregistered [ 1991.862315] Key type ._llcrypt unregistered [ 1995.915602] Key type ._llcrypt registered [ 1995.917454] Key type .llcrypt registered [ 1996.061222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.072081] alg: No test for adler32 (adler32-zlib) [ 1996.950461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1996.970893] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 2003.116699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.136@tcp [ 2009.147849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.219968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2009.247392] LNet: Added LNI 192.168.204.36@tcp1 [8/256/0/180] [ 2010.057566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2010.526852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.136@tcp [ 2010.688309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2010.709308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2010.726851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2010.746375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2010.782435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2010.820270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2010.845607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2010.872573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2010.883060] LNet: 122127:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2010.904932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2010.931247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2010.964235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2010.996545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2011.033472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2011.074396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2011.565519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2011.742363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2011.765206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2011.783956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2011.808542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2011.834272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2011.876138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2011.908423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2011.948522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2011.961523] LNet: 122244:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2011.974848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2012.007813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2012.040589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2012.081215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2012.118074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2012.156640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2012.753645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.136@tcp [ 2012.971647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2012.996893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2013.035625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2013.064927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2013.082835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2013.120368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2013.182465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2013.211688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2013.243958] LNet: 122361:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2013.278795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2013.342782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2013.361196] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.136@tcp [ 2013.386053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2013.408240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2013.435522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2013.464538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2014.020436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2014.261348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2014.292555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2014.315467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2014.341043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2014.369665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2014.394408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2014.420985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2014.449689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2014.481484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2014.509193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2014.536971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2014.559510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2014.583944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2014.628567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2018.643493] LNet: 122558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.652661] LNetError: 122558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.655753] LNet: Removed LNI 192.168.204.36@tcp [ 2018.670682] LNet: Removed LNI 192.168.204.36@tcp1 [ 2019.183193] Key type .llcrypt unregistered [ 2019.184888] Key type ._llcrypt unregistered [ 2032.791878] Key type ._llcrypt registered [ 2032.793413] Key type .llcrypt registered [ 2033.977067] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 09:41:59 (1766414519) [ 2039.765278] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 09:42:05 (1766414525) [ 2046.869364] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 09:42:12 (1766414532) [ 2052.904791] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 09:42:18 (1766414538) [ 2060.147894] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 09:42:25 (1766414545) [ 2067.308812] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 09:42:32 (1766414552) [ 2074.193865] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 09:42:39 (1766414559) [ 2081.242667] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 09:42:46 (1766414566) [ 2087.833632] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 09:42:53 (1766414573) [ 2093.966293] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 09:42:59 (1766414579) [ 2100.664833] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 09:43:06 (1766414586) [ 2107.308305] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 09:43:12 (1766414592) [ 2113.345782] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 09:43:19 (1766414599) [ 2119.153238] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 09:43:24 (1766414604) [ 2125.717458] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 09:43:31 (1766414611) [ 2132.326599] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 09:43:37 (1766414617) [ 2138.933589] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 09:43:44 (1766414624) [ 2147.214394] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 09:43:52 (1766414632) [ 2147.494556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2147.511750] alg: No test for adler32 (adler32-zlib) [ 2148.433801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.586436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.610360] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 2150.154569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.196580] LNet: 133263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2150.200506] LNetError: 133263:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2150.215486] LNet: Removed LNI 192.168.204.36@tcp [ 2150.250492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2150.314692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2150.353964] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 2158.610260] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 09:44:03 (1766414643) [ 2167.497249] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 09:44:12 (1766414652) [ 2175.113310] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 09:44:20 (1766414660) [ 2181.578609] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 09:44:27 (1766414667) [ 2188.754148] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 09:44:34 (1766414674) [ 2195.454127] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 09:44:41 (1766414681) [ 2200.980484] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 09:44:46 (1766414686) [ 2206.697977] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 09:44:52 (1766414692) [ 2213.082775] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 09:44:58 (1766414698) [ 2219.020343] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 09:45:05 (1766414705) [ 2225.584817] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 09:45:10 (1766414710) [ 2234.047784] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 09:45:19 (1766414719) [ 2239.196598] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2015 sec ========== 09:45:25 (1766414725) [ 2239.492833] LNet: 140726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.497828] LNetError: 140726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2239.511107] LNet: Removed LNI 192.168.204.36@tcp [ 2240.210845] Key type .llcrypt unregistered [ 2240.212373] Key type ._llcrypt unregistered [ 2251.152408] Key type ._llcrypt registered [ 2251.157109] Key type .llcrypt registered [ 2251.770308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2251.797750] alg: No test for adler32 (adler32-zlib) [ 2252.819938] Lustre: Lustre: Build Version: 2.17.0_RC3_1_g3717bff [ 2253.070734] LNet: Added LNI 192.168.204.36@tcp [8/256/0/180] [ 2254.719986] Key type lgssc registered [ 2255.636473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2304.126723] Lustre: Mounted lustre-client [ 2308.715206] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2319.497738] LustreError: 143174:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8da889d40800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2319.517697] LustreError: 143174:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2319.549207] Lustre: Unmounted lustre-client [ 2338.601875] Key type lgssc unregistered [ 2338.830131] LNet: 143654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2338.843707] LNetError: 143654:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2338.863416] LNet: Removed LNI 192.168.204.36@tcp [ 2339.438582] Key type .llcrypt unregistered [ 2339.439844] Key type ._llcrypt unregistered