[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 493399141 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003344] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007951] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009015] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010009] pid_max: default: 32768 minimum: 301 [ 0.011144] LSM: Security Framework initializing [ 0.013008] Yama: becoming mindful. [ 0.014037] SELinux: Initializing. [ 0.015048] *** VALIDATE selinux *** [ 0.023095] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028411] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031099] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032118] *** VALIDATE tmpfs *** [ 0.034348] *** VALIDATE proc *** [ 0.035301] *** VALIDATE cgroup *** [ 0.036008] *** VALIDATE cgroup2 *** [ 0.038258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041018] Spectre V2 : User space: Vulnerable [ 0.042007] Speculative Store Bypass: Vulnerable [ 0.045334] debug: unmapping init [mem 0xffffffffbc059000-0xffffffffbc060fff] [ 0.047166] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048665] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049025] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051012] ... generic registers: 4 [ 0.052021] ... value mask: 0000ffffffffffff [ 0.053017] ... max period: 00007fffffffffff [ 0.054016] ... fixed-purpose events: 3 [ 0.055014] ... event mask: 000000070000000f [ 0.056392] rcu: Hierarchical SRCU implementation. [ 0.058503] smp: Bringing up secondary CPUs ... [ 0.059600] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.067343] smp: Brought up 1 node, 4 CPUs [ 0.069017] smpboot: Max logical packages: 1 [ 0.070014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130014] node 0 deferred pages initialised in 58ms [ 0.133162] devtmpfs: initialized [ 0.134215] x86/mm: Memory block size: 128MB [ 0.137547] gcov: version magic: 0x41383552 [ 0.139276] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140080] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141266] pinctrl core: initialized pinctrl subsystem [ 0.142198] [ 0.142728] ************************************************************* [ 0.143021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144012] ** ** [ 0.145017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146011] ** ** [ 0.147015] ** This means that this kernel is built to expose internal ** [ 0.148013] ** IOMMU data structures, which may compromise security on ** [ 0.149013] ** your system. ** [ 0.150013] ** ** [ 0.151015] ** If you see this message and you are not debugging the ** [ 0.152013] ** kernel, report this immediately to your vendor! ** [ 0.153016] ** ** [ 0.154012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155013] ************************************************************* [ 0.156659] NET: Registered protocol family 16 [ 0.157570] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160414] cpuidle: using governor menu [ 0.162032] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.163512] PCI: Using configuration type 1 for base access [ 0.165128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172115] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.175037] cryptd: max_cpu_qlen set to 1000 [ 0.177366] ACPI: Added _OSI(Module Device) [ 0.178017] ACPI: Added _OSI(Processor Device) [ 0.179015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.180015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.184528] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.187571] ACPI: Interpreter enabled [ 0.188077] ACPI: PM: (supports S0 S3 S4 S5) [ 0.189015] ACPI: Using IOAPIC for interrupt routing [ 0.190122] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.191409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.201228] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.202047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.203026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.204087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.206304] acpiphp: Slot [2] registered [ 0.207145] acpiphp: Slot [5] registered [ 0.208155] acpiphp: Slot [6] registered [ 0.209133] acpiphp: Slot [3] registered [ 0.210110] acpiphp: Slot [4] registered [ 0.211106] acpiphp: Slot [7] registered [ 0.212149] acpiphp: Slot [8] registered [ 0.213139] acpiphp: Slot [9] registered [ 0.214345] acpiphp: Slot [10] registered [ 0.215102] acpiphp: Slot [11] registered [ 0.216111] acpiphp: Slot [12] registered [ 0.217100] acpiphp: Slot [13] registered [ 0.218092] acpiphp: Slot [14] registered [ 0.219103] acpiphp: Slot [15] registered [ 0.220147] acpiphp: Slot [16] registered [ 0.221068] acpiphp: Slot [17] registered [ 0.222154] acpiphp: Slot [18] registered [ 0.223102] acpiphp: Slot [19] registered [ 0.224097] acpiphp: Slot [20] registered [ 0.225110] acpiphp: Slot [21] registered [ 0.226157] acpiphp: Slot [22] registered [ 0.227132] acpiphp: Slot [23] registered [ 0.228140] acpiphp: Slot [24] registered [ 0.229131] acpiphp: Slot [25] registered [ 0.230190] acpiphp: Slot [26] registered [ 0.231137] acpiphp: Slot [27] registered [ 0.232129] acpiphp: Slot [28] registered [ 0.233158] acpiphp: Slot [29] registered [ 0.234117] acpiphp: Slot [30] registered [ 0.235118] acpiphp: Slot [31] registered [ 0.236068] PCI host bridge to bus 0000:00 [ 0.237022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.238037] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.239035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.241042] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.246029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.249276] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.252272] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.255271] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.259496] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.268017] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.273062] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.277026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.279021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.282019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.285212] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.288111] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.291056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.295930] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.301016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.311015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.315019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320436] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.325016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.348927] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.357291] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.373016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.380017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.398023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.407248] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.410533] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.414062] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.417419] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.419231] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.423154] iommu: Default domain type: Passthrough [ 0.426687] SCSI subsystem initialized [ 0.427153] ACPI: bus type USB registered [ 0.429181] usbcore: registered new interface driver usbfs [ 0.430079] usbcore: registered new interface driver hub [ 0.432123] usbcore: registered new device driver usb [ 0.433235] pps_core: LinuxPPS API ver. 1 registered [ 0.435012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.438063] PTP clock support registered [ 0.440075] EDAC MC: Ver: 3.0.0 [ 0.441461] PCI: Using ACPI for IRQ routing [ 0.442863] NetLabel: Initializing [ 0.443010] NetLabel: domain hash size = 128 [ 0.444010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.445127] NetLabel: unlabeled traffic allowed by default [ 0.448135] vgaarb: loaded [ 0.449234] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.451017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.458339] clocksource: Switched to clocksource kvm-clock [ 0.564471] VFS: Disk quotas dquot_6.6.0 [ 0.566224] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.568641] *** VALIDATE ramfs *** [ 0.569965] *** VALIDATE hugetlbfs *** [ 0.571277] pnp: PnP ACPI init [ 0.573477] pnp: PnP ACPI: found 6 devices [ 0.592311] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.595128] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.597248] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.599847] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.602260] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.604879] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.607877] NET: Registered protocol family 2 [ 0.610383] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.616425] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.620190] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.625853] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.629399] TCP: Hash tables configured (established 65536 bind 65536) [ 0.632561] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.635711] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.638711] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.642172] NET: Registered protocol family 1 [ 0.644909] RPC: Registered named UNIX socket transport module. [ 0.647217] RPC: Registered udp transport module. [ 0.649048] RPC: Registered tcp transport module. [ 0.650669] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.653313] NET: Registered protocol family 44 [ 0.656393] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.659100] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.662416] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.665215] PCI: CLS 0 bytes, default 64 [ 0.667400] Unpacking initramfs... [ 2.098417] debug: unmapping init [mem 0xffff94843cc64000-0xffff94843ffcffff] [ 2.102321] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.104378] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.107875] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.636173] Initialise system trusted keyrings [ 2.637501] Key type blacklist registered [ 2.639203] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.652300] zbud: loaded [ 2.656875] *** VALIDATE nfs *** [ 2.658044] *** VALIDATE nfs4 *** [ 2.659930] pstore: using deflate compression [ 2.664132] Platform Keyring initialized [ 2.776536] NET: Registered protocol family 38 [ 2.778371] Key type asymmetric registered [ 2.782291] Asymmetric key parser 'x509' registered [ 2.784419] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.787856] io scheduler mq-deadline registered [ 2.789611] io scheduler kyber registered [ 2.791422] io scheduler bfq registered [ 2.793694] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.797528] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.801569] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.805633] ACPI: Power Button [PWRF] [ 2.812331] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.820557] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.833706] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.862153] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.892637] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.897620] Non-volatile memory driver v1.3 [ 2.899393] Linux agpgart interface v0.103 [ 2.945255] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.948363] vda: detected capacity change from 0 to 74829824 [ 2.972665] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.977093] vdb: detected capacity change from 0 to 1073741824 [ 2.987948] libphy: Fixed MDIO Bus: probed [ 2.996356] usbcore: registered new interface driver usbserial_generic [ 2.998659] usbserial: USB Serial support registered for generic [ 3.001075] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.005848] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.007822] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.011715] mousedev: PS/2 mouse device common for all mice [ 3.014989] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.018725] rtc_cmos 00:05: RTC can wake from S4 [ 3.022509] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.024662] rtc_cmos 00:05: registered as rtc0 [ 3.028974] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.030350] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.032543] intel_pstate: CPU model not supported [ 3.039692] hid: raw HID events driver (C) Jiri Kosina [ 3.041919] usbcore: registered new interface driver usbhid [ 3.044141] usbhid: USB HID core driver [ 3.045888] drop_monitor: Initializing network drop monitor service [ 3.048454] Initializing XFRM netlink socket [ 3.050845] NET: Registered protocol family 10 [ 3.055713] Segment Routing with IPv6 [ 3.057881] NET: Registered protocol family 17 [ 3.060304] mpls_gso: MPLS GSO support [ 3.067151] RAS: Correctable Errors collector initialized. [ 3.070297] AVX version of gcm_enc/dec engaged. [ 3.072333] AES CTR mode by8 optimization enabled [ 3.164788] sched_clock: Marking stable (3164762731, 0)->(4143080026, -978317295) [ 3.169558] registered taskstats version 1 [ 3.172026] Loading compiled-in X.509 certificates [ 3.175187] zswap: loaded using pool lzo/zbud [ 3.209055] Key type big_key registered [ 3.222353] Key type encrypted registered [ 3.224544] ima: No TPM chip found, activating TPM-bypass! [ 3.227322] ima: Allocated hash algorithm: sha1 [ 3.229374] ima: No architecture policies found [ 3.231195] evm: Initialising EVM extended attributes: [ 3.233350] evm: security.selinux [ 3.234673] evm: security.ima [ 3.235807] evm: security.capability [ 3.237043] evm: HMAC attrs: 0x1 [ 3.244646] rtc_cmos 00:05: setting system clock to 2026-08-24 13:44:30 UTC (1787579070) [ 3.252481] debug: unmapping init [mem 0xffffffffbd003000-0xffffffffbd1fffff] [ 3.256227] debug: unmapping init [mem 0xffffffffbbd82000-0xffffffffbc058fff] [ 3.266060] Write protecting the kernel read-only data: 28672k [ 3.269409] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 3.272623] debug: unmapping init [mem 0xffffffffbad14000-0xffffffffbadfffff] [ 3.327353] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.337240] systemd[1]: Detected virtualization kvm. [ 3.338843] systemd[1]: Detected architecture x86-64. [ 3.342617] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.368505] systemd[1]: No hostname configured. [ 3.370298] systemd[1]: Set hostname to . [ 3.372606] random: systemd: uninitialized urandom read (16 bytes read) [ 3.375394] systemd[1]: Initializing machine ID from random generator. [ 3.419901] random: ln: uninitialized urandom read (6 bytes read) [ 3.518753] random: systemd: uninitialized urandom read (16 bytes read) [ 3.521854] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.528117] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.532659] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.227055] device-mapper: uevent: version 1.0.3 [ 4.229491] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.092502] virtio_net virtio0 ens2: renamed from eth0 [ 5.127210] scsi host0: ata_piix [ 5.155612] scsi host1: ata_piix [ 5.159306] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.162309] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 7.239807] random: fast init done [ 9.507221] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.746608] random: crng init done [ 9.748325] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.239148] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.555748] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.844555] SELinux: Disabled at runtime. [ 11.913721] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.925331] systemd[1]: Detected virtualization kvm. [ 11.929559] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.521857] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.528650] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.535152] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.541399] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.546912] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.559372] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.571688] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ 12.669241] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.019493] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.479540] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.488707] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.643319] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.664277] EDAC sbridge: Ver: 1.1.2 [ 15.240201] Key type dns_resolver registered [ 15.561063] NFS: Registering the id_resolver key type [ 15.563399] Key type id_resolver registered [ 15.565825] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg433-client login: [ 46.328110] libcfs: loading out-of-tree module taints kernel. [ 46.426428] Key type ._llcrypt registered [ 46.428044] Key type .llcrypt registered [ 46.715940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 46.724118] alg: No test for adler32 (adler32-zlib) [ 48.378364] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 49.349559] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 51.184635] Key type lgssc registered [ 53.062496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 93.613010] hrtimer: interrupt took 7130679 ns [ 199.390264] Lustre: Mounted lustre-client - version 2.17.57_80_g35a8f2d [ 204.573196] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 218.859414] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing check_logdir /tmp/testlogs/ [ 224.761221] Lustre: DEBUG MARKER: oleg433-client.virtnet: executing yml_node [ 225.248524] Lustre: lustre-OST0000-osc-ffff948489d6b000: disconnect after 23s idle [ 228.452459] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 231.398345] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 234.242799] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 236.146928] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Aug 24 09:48:21 EDT 2026 [ 251.796114] Lustre: Unmounted lustre-client [ 279.482664] Key type lgssc unregistered [ 279.765105] LNet: 8430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.770536] LNetError: 8430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 279.792633] LNet: Removed LNI 192.168.204.33@tcp [ 280.585446] Key type .llcrypt unregistered [ 280.588778] Key type ._llcrypt unregistered [ 308.010886] Key type ._llcrypt registered [ 308.016153] Key type .llcrypt registered [ 308.312101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 308.325611] alg: No test for adler32 (adler32-zlib) [ 309.273923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 309.321213] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 309.338989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 313.085819] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 313.428796] LNet: 9320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.438896] LNetError: 9320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 313.454726] LNet: Removed LNI 192.168.204.33@tcp [ 314.160622] Key type .llcrypt unregistered [ 314.162793] Key type ._llcrypt unregistered [ 322.901592] Key type ._llcrypt registered [ 322.904280] Key type .llcrypt registered [ 324.668165] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:49:50 (1787579390) [ 324.932134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 324.946799] alg: No test for adler32 (adler32-zlib) [ 325.853884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.948364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.984886] LNet: 9735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.005642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.047751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 326.064733] LNet: 9742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 332.403907] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:49:58 (1787579398) [ 332.551714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.588042] LNet: 10319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.613943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.701990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 338.640095] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:50:04 (1787579404) [ 338.806941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.859625] LNet: 10901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.885366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.977831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 345.119867] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:50:10 (1787579410) [ 345.311186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.353561] LNet: 11489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.427588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.552657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 353.055145] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:50:18 (1787579418) [ 353.181015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.231060] LNet: 12071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.266836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.380715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 353.730222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 359.745401] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:50:25 (1787579425) [ 359.965037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.020472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.092914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.377470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 360.790481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 367.197693] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:50:33 (1787579433) [ 367.274556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.320050] LNet: 13254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.329238] LNet: 13254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 367.367227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.439442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 374.814824] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:50:40 (1787579440) [ 374.929711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.079875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.281820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 375.326438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 376.082871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 376.154953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 377.644782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 377.719442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.295342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 378.363227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 378.954358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 378.979605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 379.333932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 379.355884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 379.626312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 379.662767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.133491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 380.180888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 380.700771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 380.794042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 381.464738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 381.496535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 382.155800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 382.258409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 383.274670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 383.383866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 384.196321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 384.282312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 394.145742] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:50:59 (1787579459) [ 394.384953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.400643] LNet: 14530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.415855] LNet: 14530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.442618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.503338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 394.534619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 400.913996] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:51:06 (1787579466) [ 401.116235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.136951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.207887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 401.242490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 408.097802] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:51:13 (1787579473) [ 408.234189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.282694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.411729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 408.451353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 416.010588] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:51:21 (1787579481) [ 416.101690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.172047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.278913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 416.315348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 423.284768] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:51:28 (1787579488) [ 423.659342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.709563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.809631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 423.872532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 431.598383] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:51:37 (1787579497) [ 431.714311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.735108] LNet: 17455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.741556] LNet: 17455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 431.783256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.934436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 431.976838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 439.721400] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:51:45 (1787579505) [ 439.817216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.880980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.993209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 440.593703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.668853] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:51:53 (1787579513) [ 448.028372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.087836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.353199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 448.596506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 455.330302] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:52:01 (1787579521) [ 455.473293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.611685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.833713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.138281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.209596] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:52:08 (1787579528) [ 462.383931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.404664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.905512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.117381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.582897] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:52:17 (1787579537) [ 471.699257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.754360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.872063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.926882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.701955] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:52:24 (1787579544) [ 478.950898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 478.979661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.043079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.091825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 485.605561] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:52:31 (1787579551) [ 485.816667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.879091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.930486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 492.952558] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:52:38 (1787579558) [ 493.072707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.158708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.282094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 499.781266] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:52:45 (1787579565) [ 499.856570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.906417] LNet: 22891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 499.915719] LNet: 22891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 499.943977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.054240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 500.529279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 507.083338] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:52:52 (1787579572) [ 507.307432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.348208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.448905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 507.489226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 513.651726] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:52:59 (1787579579) [ 513.941309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.972745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 514.049611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 514.096056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 520.249942] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:53:06 (1787579586) [ 520.453452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 520.527702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 520.574321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 520.611814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 527.852207] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:53:13 (1787579593) [ 527.939932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.018168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 528.144801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 528.218528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 534.751079] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:53:20 (1787579600) [ 534.868416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.951887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.058949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 535.119199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 535.152743] LNetError: 25845:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 535.194826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 535.236178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 541.293483] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:53:27 (1787579607) [ 541.530208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.571357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.622308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.46QtXqmG2n [ 541.661037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.46QtXqmG2n [ 541.701125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.46QtXqmG2n [ 549.570148] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:53:34 (1787579614) [ 549.964559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 550.086936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 550.235627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 550.296076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 550.349741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 558.822482] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:53:44 (1787579624) [ 558.961579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 559.031437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 559.170627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 559.205865] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 561.442631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 562.985405] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 564.856257] Key type lgssc registered [ 566.344389] Lustre: Echo OBD driver; http://www.lustre.org/ [ 569.787127] LNetError: 27606:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.33@tcp99 has no available nets [ 572.488471] Key type lgssc unregistered [ 572.983625] LNetError: 28718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.008121] LNet: Removed LNI 192.168.204.33@tcp [ 573.579600] Key type .llcrypt unregistered [ 573.586174] Key type ._llcrypt unregistered [ 582.014800] Key type ._llcrypt registered [ 582.020093] Key type .llcrypt registered [ 584.053104] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:54:09 (1787579649) [ 584.710879] Key type .llcrypt unregistered [ 584.722264] Key type ._llcrypt unregistered [ 593.689994] Key type ._llcrypt registered [ 593.691352] Key type .llcrypt registered [ 593.970146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 594.003369] alg: No test for adler32 (adler32-zlib) [ 595.006920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.091681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 595.109887] LNet: 29641:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 595.115745] LNet: 29641:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 595.121266] LNet: 29641:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 595.131744] LNet: Message forwarding will be enabled in 2s [ 598.242303] LNet: Message forwarding enabled [ 598.733266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 598.753880] LNet: Message forwarding will stop in 2s to 6s [ 602.338959] LNet: Message forwarding disabled [ 603.549457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 603.609986] LNet: 29739:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 603.623773] LNet: 29739:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 603.637976] LNet: 29739:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 603.669520] LNet: Message forwarding will be enabled in 2s [ 603.944143] LNet: 29752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.622563] Key type .llcrypt unregistered [ 604.626078] Key type ._llcrypt unregistered [ 618.432402] Key type ._llcrypt registered [ 618.434109] Key type .llcrypt registered [ 619.933984] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:54:45 (1787579685) [ 620.465198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.481653] alg: No test for adler32 (adler32-zlib) [ 621.404901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.477485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 628.915862] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:54:54 (1787579694) [ 629.033028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.079833] LNet: 31154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.112843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.233774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 629.277216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 629.325564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 629.381949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 629.433810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 629.487888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 629.562908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 629.624485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 629.664888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 629.719923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 629.749831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 629.787968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 629.843104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 629.901934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 629.942489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 630.001833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 630.059122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 630.132451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.173826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 630.206400] LNetError: 31203:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 637.009328] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:55:02 (1787579702) [ 637.104889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.144535] LNet: 31770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.170374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.270250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 643.934410] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:55:09 (1787579709) [ 646.050210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg433-server [ 647.792697] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:55:13 (1787579713) [ 647.917755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.948242] LNet: 32702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.984378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.061941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.089750] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 649.031863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[34-38]@tcp [ 649.053890] LNet: 32839:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 649.119317] LNetError: 32710:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.34@tcp is being used as a gateway but routing feature is not turned on [ 649.145063] LNetError: 32710:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.34@tcp has gone from up to down [ 656.144442] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:55:22 (1787579722) [ 657.954283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg433-server [ 659.840454] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:55:25 (1787579725) [ 660.038032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 660.087673] LNet: 33774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.094955] LNetError: 33774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.174497] LNet: Removed LNI 192.168.204.33@tcp [ 661.281639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.390205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.425642] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 662.113694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[34-39/2]@tcp [ 662.150061] LNet: 33894:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 662.165771] LNet: 33894:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 662.225944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[34-39/2]@tcp [ 669.330954] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:55:34 (1787579734) [ 669.455553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.507515] LNet: 34480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.518351] LNetError: 34480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.548926] LNet: Removed LNI 192.168.204.33@tcp [ 669.627365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.808477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 669.893320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.012864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.131891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.252420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.331786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.424316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 670.482925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.522591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.579275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.675708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.705424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.759813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 670.850342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.900399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.964852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.069441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.129290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.190699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 671.292518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.359755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.452357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.553933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.605443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.664321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.751257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.798801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.892220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.963288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.001837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.056732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 672.139104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.203262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.285346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 679.275398] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:55:45 (1787579745) [ 681.348667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg433-server [ 683.312628] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:55:49 (1787579749) [ 685.686597] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg433-server [ 687.666668] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:55:53 (1787579753) [ 687.773040] LNet: 35879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.789570] LNet: 35879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 688.475275] Key type .llcrypt unregistered [ 688.478734] Key type ._llcrypt unregistered [ 693.767238] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 693.992479] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 694.004691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 694.013177] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 696.529961] Key type ._llcrypt registered [ 696.532027] Key type .llcrypt registered [ 696.805476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 696.844787] alg: No test for adler32 (adler32-zlib) [ 697.744185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.804729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.836789] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 698.466916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 698.489543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 700.444984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 700.463349] LNet: Removed LNI 10.1.2.3@tcp [ 700.585662] LNet: 36606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 700.601519] LNetError: 36606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 700.630786] LNet: Removed LNI 192.168.204.33@tcp [ 701.303533] Key type .llcrypt unregistered [ 701.306859] Key type ._llcrypt unregistered [ 713.343814] Key type ._llcrypt registered [ 713.345728] Key type .llcrypt registered [ 714.744515] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:56:20 (1787579780) [ 715.564314] Key type .llcrypt unregistered [ 715.571601] Key type ._llcrypt unregistered [ 722.143685] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 722.230359] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.235214] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.239028] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 723.968624] Key type ._llcrypt registered [ 723.978506] Key type .llcrypt registered [ 724.183653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.195631] alg: No test for adler32 (adler32-zlib) [ 725.231185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.288372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 725.321659] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 726.182154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 726.210302] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 728.869958] LNet: 38102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.874509] LNetError: 38102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 728.882843] LNet: Removed LNI 10.1.2.3@tcp [ 728.905410] LNet: Removed LNI 192.168.204.33@tcp [ 729.721885] Key type .llcrypt unregistered [ 729.726063] Key type ._llcrypt unregistered [ 743.058448] Key type ._llcrypt registered [ 743.060466] Key type .llcrypt registered [ 744.411607] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:56:50 (1787579810) [ 745.143331] Key type .llcrypt unregistered [ 745.149273] Key type ._llcrypt unregistered [ 752.120639] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.262959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.269316] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.275483] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.953801] Key type ._llcrypt registered [ 754.955091] Key type .llcrypt registered [ 755.131777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.147528] alg: No test for adler32 (adler32-zlib) [ 756.103341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.210496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 756.267645] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 758.213748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 758.234378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 760.420825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 760.436415] LNet: Removed LNI 10.1.2.3@tcp [ 760.448932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 760.488707] LNet: Removed LNI 10.1.2.31@tcp [ 760.502134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 760.531417] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.235389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 761.257236] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 762.185696] LNet: 40048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.193959] LNetError: 40048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 762.200519] LNet: Removed LNI 10.1.2.31@tcp [ 762.211491] LNet: Removed LNI 10.1.2.3@tcp [ 762.918471] Key type .llcrypt unregistered [ 762.926324] Key type ._llcrypt unregistered [ 774.192964] Key type ._llcrypt registered [ 774.195642] Key type .llcrypt registered [ 775.545223] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:57:21 (1787579841) [ 776.011095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 776.027561] alg: No test for adler32 (adler32-zlib) [ 776.961581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.006976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 777.038423] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 777.692327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 777.709298] LNet: 41004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.717823] LNetError: 41004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.728805] LNet: Removed LNI 192.168.204.33@tcp [ 777.753693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.845807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.33@tcp [ 777.874755] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 778.538922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.33@tcp [ 778.566561] LNet: Removed LNI 192.168.204.33@tcp [ 778.632827] LNet: 41139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.252532] Key type .llcrypt unregistered [ 779.255901] Key type ._llcrypt unregistered [ 788.433385] Key type ._llcrypt registered [ 788.439281] Key type .llcrypt registered [ 789.783671] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:57:35 (1787579855) [ 791.319853] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg433-server [ 793.028273] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:57:38 (1787579858) [ 793.609558] Key type .llcrypt unregistered [ 793.615152] Key type ._llcrypt unregistered [ 801.073367] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 801.257411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 801.272228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 801.281020] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 804.051125] Key type ._llcrypt registered [ 804.057713] Key type .llcrypt registered [ 804.315320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 804.339651] alg: No test for adler32 (adler32-zlib) [ 805.349888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.458960] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 806.487636] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 809.330873] LNet: 42979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.349512] LNetError: 42979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.357244] LNet: Removed LNI 192.168.204.33@tcp1 [ 809.392503] LNet: Removed LNI 10.1.2.3@tcp2 [ 810.342380] Key type .llcrypt unregistered [ 810.345146] Key type ._llcrypt unregistered [ 821.666484] Key type ._llcrypt registered [ 821.668111] Key type .llcrypt registered [ 823.239520] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:58:08 (1787579888) [ 823.641774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.666432] alg: No test for adler32 (adler32-zlib) [ 824.633628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.679665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.711601] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 826.023984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.067402] LNet: 43955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.077189] LNetError: 43955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.092911] LNet: Removed LNI 192.168.204.33@tcp [ 826.106624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.213844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 826.239577] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 832.735901] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:58:18 (1787579898) [ 832.845904] LNet: 44542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.858163] LNetError: 44542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.875098] LNet: Removed LNI 192.168.204.33@tcp [ 833.464072] Key type .llcrypt unregistered [ 833.467926] Key type ._llcrypt unregistered [ 839.536071] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.645367] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.658501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.666638] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.270795] Key type ._llcrypt registered [ 842.279218] Key type .llcrypt registered [ 842.447714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.458528] alg: No test for adler32 (adler32-zlib) [ 843.422721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.521157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.580680] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 844.440760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 844.462111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 846.361123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.389568] LNet: 45292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.399598] LNetError: 45292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 846.408515] LNet: Removed LNI 10.1.2.3@tcp [ 846.417478] LNet: Removed LNI 192.168.204.33@tcp [ 846.444643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.545858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 846.594636] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 846.604171] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 847.145484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.165368] LNet: 45319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.175568] LNetError: 45319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 847.183157] LNetError: 45319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 847.191516] LNet: Removed LNI 10.1.2.3@tcp [ 847.204210] LNet: Removed LNI 192.168.204.33@tcp [ 847.240950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.304225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 847.332031] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 847.337660] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 847.905461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 847.928081] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 849.037203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.064088] LNet: 45481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.083548] LNetError: 45481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 849.101516] LNetError: 45481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 849.117373] LNet: Removed LNI 10.1.2.3@tcp [ 849.131559] LNet: Removed LNI 192.168.204.33@tcp [ 849.147530] LNet: Removed LNI 192.168.204.33@tcp2 [ 849.189546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.278447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 849.318556] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 849.327992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 849.334951] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 857.122546] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:58:42 (1787579922) [ 857.219088] LNet: 46069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.239134] LNetError: 46069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 857.261167] LNetError: 46069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 857.279068] LNet: Removed LNI 10.1.2.3@tcp [ 857.282296] LNet: Removed LNI 192.168.204.33@tcp [ 857.312466] LNet: Removed LNI 192.168.204.33@tcp2 [ 858.069923] Key type .llcrypt unregistered [ 858.073885] Key type ._llcrypt unregistered [ 863.246618] Key type ._llcrypt registered [ 863.248333] Key type .llcrypt registered [ 863.406611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 863.417746] alg: No test for adler32 (adler32-zlib) [ 864.316718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.395460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 864.438704] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 865.201248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.243947] LNet: 46476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.254280] LNetError: 46476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.281616] LNet: Removed LNI 192.168.204.33@tcp [ 865.298960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.360270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 865.386446] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 866.026379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.040960] LNet: 46506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.045547] LNetError: 46506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.067618] LNet: Removed LNI 192.168.204.33@tcp [ 866.087487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.146970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 866.161849] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 866.436813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.461183] LNet: Removed LNI 192.168.204.33@tcp [ 866.481890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.545668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 866.571841] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 867.010118] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 867.151204] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 867.154385] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 867.157598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 869.509332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 869.525890] LNet: 46611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.531476] LNet: 46611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 869.551901] LNetError: 46611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.564784] LNetError: 46611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 869.589079] LNet: Removed LNI 192.168.204.33@tcp [ 869.640302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.700702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 869.737194] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 877.126135] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:59:02 (1787579942) [ 877.270366] LNet: 47202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.290694] LNetError: 47202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.343583] LNet: Removed LNI 192.168.204.33@tcp [ 878.216949] Key type .llcrypt unregistered [ 878.222411] Key type ._llcrypt unregistered [ 886.182707] Key type ._llcrypt registered [ 886.190750] Key type .llcrypt registered [ 886.554969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.576863] alg: No test for adler32 (adler32-zlib) [ 887.473464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.528279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.558440] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 888.303932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 888.340143] LNet: 47611:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 888.348852] LNetError: 47611:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 888.381156] LNet: Removed LNI 192.168.204.33@tcp [ 888.426792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 888.528382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 888.566788] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 896.335602] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:59:22 (1787579962) [ 896.719649] LNet: 48203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.743330] LNetError: 48203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.767654] LNet: Removed LNI 192.168.204.33@tcp [ 897.495750] Key type .llcrypt unregistered [ 897.498067] Key type ._llcrypt unregistered [ 904.819457] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 905.042666] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 905.063899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 905.072831] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 906.650336] Key type ._llcrypt registered [ 906.651939] Key type .llcrypt registered [ 906.886593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.903359] alg: No test for adler32 (adler32-zlib) [ 907.843192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.933646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 907.968711] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 908.782345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 908.796489] LNet: 48650:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.800960] LNetError: 48650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.814856] LNet: Removed LNI 192.168.204.33@tcp [ 908.833983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.882557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 908.914262] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 915.770634] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:59:41 (1787579981) [ 915.891901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.931870] LNet: 49241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.938914] LNetError: 49241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 915.958991] LNet: Removed LNI 192.168.204.33@tcp [ 916.000574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.119691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 922.770451] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:59:48 (1787579988) [ 922.881890] LNet: 49824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.880274] Key type .llcrypt unregistered [ 923.881527] Key type ._llcrypt unregistered [ 930.757794] Key type ._llcrypt registered [ 930.760338] Key type .llcrypt registered [ 931.145263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.182389] alg: No test for adler32 (adler32-zlib) [ 932.124778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.256574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.329697] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 933.399801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.420389] LNet: 50232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.425949] LNetError: 50232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.444442] LNet: Removed LNI 192.168.204.33@tcp [ 933.487185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.584750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 933.612205] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 941.715380] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:00:07 (1787580007) [ 941.923535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 941.947958] LNet: 50825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.960568] LNetError: 50825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.979811] LNet: Removed LNI 192.168.204.33@tcp [ 942.045030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.208059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 948.825971] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:00:14 (1787580014) [ 950.594665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 952.420734] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:00:18 (1787580018) [ 954.165899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 955.994572] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:00:21 (1787580021) [ 957.665975] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 959.505397] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:00:25 (1787580025) [ 961.000309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 962.597031] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:00:28 (1787580028) [ 964.125842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 965.720704] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:00:31 (1787580031) [ 965.834221] LNet: 52869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 966.526645] Key type .llcrypt unregistered [ 966.529792] Key type ._llcrypt unregistered [ 973.558433] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 973.685695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 973.694467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 973.703094] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 976.291714] Key type ._llcrypt registered [ 976.294656] Key type .llcrypt registered [ 976.613717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.653968] alg: No test for adler32 (adler32-zlib) [ 977.695378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.861513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 977.901238] LNet: Added LNI 192.168.204.33@tcp [8/512/0/180] [ 977.911801] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 977.917963] LNet: 53205:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.064778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.085192] LNet: 53224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.091899] LNetError: 53224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.098642] LNet: Removed LNI 10.1.2.3@tcp [ 978.106218] LNetError: 53224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.128234] LNet: Removed LNI 192.168.204.33@tcp [ 978.192206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.282792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.314987] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 978.389525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.432282] LNet: 53248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.451549] LNetError: 53248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.466162] LNet: Removed LNI 192.168.204.33@tcp [ 978.490122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.552336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.580193] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 978.588978] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 978.595959] LNet: 53256:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 978.716218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 978.748960] LNet: 53274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.755891] LNetError: 53274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 978.763120] LNet: Removed LNI 10.1.2.3@tcp [ 978.765508] LNetError: 53274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.780483] LNet: Removed LNI 192.168.204.33@tcp [ 978.802459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.939156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 978.980426] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 978.985928] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 978.987916] LNet: 53282:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 979.200775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.225753] LNet: 53308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.232119] LNetError: 53308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.239836] LNet: Removed LNI 10.1.2.3@tcp [ 979.242886] LNetError: 53308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.256778] LNet: Removed LNI 192.168.204.33@tcp [ 979.279662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.361814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 979.386860] LNet: Added LNI 192.168.204.33@tcp [8/256/0/182] [ 979.391164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 979.530595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.569348] LNet: 53336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.580091] LNetError: 53336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 979.592728] LNet: Removed LNI 10.1.2.3@tcp [ 979.597348] LNetError: 53336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.608828] LNet: Removed LNI 192.168.204.33@tcp [ 979.629947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.760838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 979.794223] LNet: Added LNI 192.168.204.33@tcp [8/256/0/182] [ 979.799861] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 979.801832] LNet: 53344:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 979.886868] LNet: Removed LNI 10.1.2.3@tcp [ 979.904912] LNet: Removed LNI 192.168.204.33@tcp [ 980.673424] Key type .llcrypt unregistered [ 980.678444] Key type ._llcrypt unregistered [ 992.934759] Key type ._llcrypt registered [ 992.937496] Key type .llcrypt registered [ 994.582918] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:01:00 (1787580060) [ 996.281667] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 998.251716] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:01:04 (1787580064) [ 998.609377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.625975] alg: No test for adler32 (adler32-zlib) [ 999.564675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 999.640897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 999.667409] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1000.455890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.33 [ 1000.491332] LNet: 54627:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1000.514163] LNet: Removed LNI 192.168.204.33@tcp [ 1006.894907] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:01:12 (1787580072) [ 1007.092961] LNet: 55195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.104110] LNetError: 55195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1007.128677] LNet: Removed LNI 192.168.204.33@tcp [ 1007.788603] Key type .llcrypt unregistered [ 1007.791833] Key type ._llcrypt unregistered [ 1014.304063] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1014.551225] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1017.315304] Key type ._llcrypt registered [ 1017.317151] Key type .llcrypt registered [ 1017.482289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1017.499883] alg: No test for adler32 (adler32-zlib) [ 1018.856461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1026.095467] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:01:31 (1787580091) [ 1026.239485] LNet: 56107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1026.262149] LNetError: 56107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1026.293334] LNet: Removed LNI 10.1.2.3@tcp [ 1026.892325] Key type .llcrypt unregistered [ 1026.894809] Key type ._llcrypt unregistered [ 1033.512713] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1033.743959] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1033.753913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1033.767234] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1036.635170] Key type ._llcrypt registered [ 1036.637393] Key type .llcrypt registered [ 1036.815493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1036.824993] alg: No test for adler32 (adler32-zlib) [ 1037.778616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.321286] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:01:50 (1787580110) [ 1046.100467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1047.634758] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:01:53 (1787580113) [ 1047.795161] LNet: 57298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.807255] LNetError: 57298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1047.831993] LNet: Removed LNI 10.1.2.3@tcp [ 1048.535777] Key type .llcrypt unregistered [ 1048.538888] Key type ._llcrypt unregistered [ 1053.957179] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1054.142404] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1054.147668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.154967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1056.873606] Key type ._llcrypt registered [ 1056.877376] Key type .llcrypt registered [ 1057.069805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1057.080304] alg: No test for adler32 (adler32-zlib) [ 1058.035918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1058.106238] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1065.481874] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:02:11 (1787580131) [ 1065.866521] LNet: 58538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.873648] LNetError: 58538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1065.901577] LNet: Removed LNI 10.1.2.3@tcp [ 1066.564765] Key type .llcrypt unregistered [ 1066.569172] Key type ._llcrypt unregistered [ 1072.545334] Key type ._llcrypt registered [ 1072.547843] Key type .llcrypt registered [ 1072.785249] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.798721] alg: No test for adler32 (adler32-zlib) [ 1073.724508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1073.774488] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1081.552113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1082.324142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1082.383538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1082.455641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1082.488634] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1082.857687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1082.894767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1082.939678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1082.979650] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1083.001190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1083.029894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1083.080655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1083.121744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1083.178277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1083.210341] LNet: 58980:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1083.252096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1083.290141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1083.341934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1083.376965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.419907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1083.479505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1083.505872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1084.093358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1084.162287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1084.216509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1084.248084] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1084.546382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1084.575863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1084.605354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1084.641416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1084.672444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1084.700866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1084.749328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1084.781631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1084.798915] LNet: 59089:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1084.824195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1084.861149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1084.894260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1084.946573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1085.011510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1085.093495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1085.150519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1085.658698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1085.704261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1085.754598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1085.772843] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1085.779477] LNet: Skipped 1 previous similar message [ 1086.086684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1086.142831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1086.210158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1086.264575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1086.303123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1086.345905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1086.391967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.431459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.455561] LNet: 59199:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.474211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.504619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.536672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.589540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.629809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.675233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.732611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1087.229675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1087.267596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1087.302478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1087.576407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.637633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.686348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.718773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.752724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.802574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.859231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.896484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.940639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.963783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.995233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1088.021627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1088.053657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.094772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.123754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.666711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1088.718701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1088.761025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1088.792941] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1088.798647] LNet: Skipped 1 previous similar message [ 1088.982556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.005871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.046540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.089758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.132339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.160492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.199415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.239894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.265709] LNet: 59417:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.277522] LNet: 59417:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1089.307231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.347812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.375272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.404580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.442831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.489922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.526403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.015175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1090.060954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1090.112675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1090.371676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.422026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.466554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.534663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1090.595952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1090.685202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1090.758363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1090.822701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1090.876935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1090.930610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1090.975102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.030056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.075708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.121585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.172732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1091.510981] LNet: 59571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.524449] LNetError: 59571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.548135] LNet: Removed LNI 192.168.204.33@tcp [ 1091.972209] Key type .llcrypt unregistered [ 1091.978234] Key type ._llcrypt unregistered [ 1109.727878] Key type ._llcrypt registered [ 1109.730635] Key type .llcrypt registered [ 1110.826994] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:02:56 (1787580176) [ 1111.824360] Key type .llcrypt unregistered [ 1111.827870] Key type ._llcrypt unregistered [ 1117.995739] Key type ._llcrypt registered [ 1118.000298] Key type .llcrypt registered [ 1118.220760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1118.238482] alg: No test for adler32 (adler32-zlib) [ 1119.223408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1119.243602] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1126.405028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1136.887467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.942966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1136.980979] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1138.250901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1138.296327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 1 [ 1138.336120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1138.382789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 1 [ 1138.456890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_interrupt -r 1 [ 1138.491450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 1 [ 1138.536364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_interrupt -r 1 [ 1138.565888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 1 [ 1138.605409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1138.632969] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1138.643644] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1138.901445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1138.933391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1138.967794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1138.993624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1139.015656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1139.056201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1139.140669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1139.222034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1139.272675] LNet: 61045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1139.315356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1139.341946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1139.369346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1139.395608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.423424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1139.461497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1139.512229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1140.026558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1140.056898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 1 [ 1140.089877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1140.117835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 1 [ 1140.144975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_dropped -r 1 [ 1140.183931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 1 [ 1140.238743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_dropped -r 1 [ 1140.300950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 1 [ 1140.346827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1140.370336] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1140.379824] LNet: Skipped 2 previous similar messages [ 1140.607932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1140.639758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1140.670641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1140.698456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1140.726841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1140.750981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1140.779309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1140.809933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1140.824495] LNet: 61166:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1140.845335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1140.871855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1140.904395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1140.930940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1140.952563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1140.983926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1141.017451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1141.442629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1141.488977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 1 [ 1141.540554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1141.575883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 1 [ 1141.625893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_aborted -r 1 [ 1141.675262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 1 [ 1141.715526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_aborted -r 1 [ 1141.773254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 1 [ 1141.843594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1141.876546] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1141.883555] LNet: Skipped 2 previous similar messages [ 1142.105420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1142.138468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1142.183767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1142.212618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1142.242566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1142.276385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1142.307431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1142.345480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1142.363319] LNet: 61287:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1142.399339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1142.449362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1142.499265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1142.533669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1142.579855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1142.636248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1142.670940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1143.157550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1143.206516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 1 [ 1143.248050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1143.298193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 1 [ 1143.331462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_no_route -r 1 [ 1143.369840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 1 [ 1143.396296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_no_route -r 1 [ 1143.423371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 1 [ 1143.459792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1143.716753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1143.743833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1143.767536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1143.798495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1143.828234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1143.869825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1143.908750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1143.946743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1144.014689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1144.048935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1144.085238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1144.111865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1144.149962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1144.175593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1144.208659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1144.748378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1144.786521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 1 [ 1144.853615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1144.893622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 1 [ 1144.928036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_timeout -r 1 [ 1144.962821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 1 [ 1144.993940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_timeout -r 1 [ 1145.015592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 1 [ 1145.040812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1145.053696] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1145.057648] LNet: Skipped 5 previous similar messages [ 1145.233688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1145.259964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1145.286462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1145.317978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1145.344490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1145.364925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1145.390924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1145.414148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1145.430396] LNet: 61530:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1145.436527] LNet: 61530:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1145.454799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1145.478913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1145.510220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1145.549362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1145.581694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1145.611266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1145.638913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1146.220696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1146.252746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_error -r 1 [ 1146.278420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1146.310464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_error -r 1 [ 1146.337755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_error -r 1 [ 1146.362990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_error -r 1 [ 1146.391352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_error -r 1 [ 1146.426940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_error -r 1 [ 1146.454453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1146.712652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1146.760605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1146.787107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1146.845473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1146.877658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1146.925851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1146.975249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1147.031218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1147.071687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1147.112148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1147.158146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1147.187512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1147.215289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1147.244694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1147.280383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1152.833899] LNet: 61731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.838879] LNetError: 61731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.848306] LNet: Removed LNI 192.168.204.33@tcp [ 1153.897737] LNet: Removed LNI 192.168.204.33@tcp1 [ 1154.367980] Key type .llcrypt unregistered [ 1154.371670] Key type ._llcrypt unregistered [ 1170.680344] Key type ._llcrypt registered [ 1170.683220] Key type .llcrypt registered [ 1171.908075] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:03:57 (1787580237) [ 1172.742180] Key type .llcrypt unregistered [ 1172.746494] Key type ._llcrypt unregistered [ 1179.178823] Key type ._llcrypt registered [ 1179.181826] Key type .llcrypt registered [ 1179.399089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1179.424281] alg: No test for adler32 (adler32-zlib) [ 1180.310375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1180.359186] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1188.291338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1188.997152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1189.029324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1189.059933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1189.073624] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1189.292572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.319524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.345419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.374768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.407143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.434300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.472448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.516570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.538763] LNet: 62986:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.558766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.590416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.617510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1189.669188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1189.725272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1189.801389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.842375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1190.471156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1190.555877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1190.588224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1190.612325] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1190.853751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1190.881901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1190.916482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1190.956772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.995358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.032167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.086489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.131421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.151521] LNet: 63095:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.166281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.189859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.213351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.242322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.276895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.308653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.347363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.857945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1191.903862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1191.947609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1191.964875] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1192.165936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.198285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.224100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.256174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.290607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.322338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.358905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.395357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.415255] LNet: 63204:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1192.441171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.477957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.513334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.550824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.593265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.651047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.684768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1193.042108] LNet: 63249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.055854] LNetError: 63249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.092769] LNet: Removed LNI 192.168.204.33@tcp [ 1193.592491] Key type .llcrypt unregistered [ 1193.596388] Key type ._llcrypt unregistered [ 1211.093935] Key type ._llcrypt registered [ 1211.096255] Key type .llcrypt registered [ 1212.176705] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:04:38 (1787580278) [ 1212.902304] Key type .llcrypt unregistered [ 1212.907445] Key type ._llcrypt unregistered [ 1219.754225] Key type ._llcrypt registered [ 1219.755774] Key type .llcrypt registered [ 1220.026257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1220.054398] alg: No test for adler32 (adler32-zlib) [ 1220.980873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1221.041654] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1229.016738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1238.454419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1238.516771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.531353] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1239.796523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1239.847931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 1 [ 1239.908594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1239.950256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 1 [ 1240.017546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_dropped -r 1 [ 1240.069700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 1 [ 1240.135921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_dropped -r 1 [ 1240.189410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 1 [ 1240.235352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1240.252329] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1240.259894] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1240.531733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.565928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.605937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.643378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.677544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.716139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.775649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.824334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.861745] LNet: 64723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.891893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.950904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.004323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.061469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.112657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.186285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.233175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1241.282682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.858622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1241.903668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_error -r 1 [ 1241.958231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1242.011994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_error -r 1 [ 1242.060482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_error -r 1 [ 1242.126487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_error -r 1 [ 1242.172490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_error -r 1 [ 1242.218186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_error -r 1 [ 1242.272670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1242.301966] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1242.315647] LNet: Skipped 2 previous similar messages [ 1242.556516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.591748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.625651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.667033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.705538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.752517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.797841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1242.836704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1242.854585] LNet: 64846:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1242.876460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1242.920804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1242.958665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1242.981988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.007557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.040528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.087819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1243.141570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.609863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1243.646783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_timeout -r 1 [ 1243.681395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1243.715245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_timeout -r 1 [ 1243.758301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_timeout -r 1 [ 1243.817648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_timeout -r 1 [ 1243.868072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_timeout -r 1 [ 1243.899655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_timeout -r 1 [ 1243.969620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.133@tcp [ 1244.001263] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1244.270250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.296879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.345437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.383142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.433444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.474393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.528693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.583682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.617939] LNet: 64969:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.642903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.691543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.734717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.787076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.818547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.870225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.929385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.970853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.053831] LNet: 65051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1250.064062] LNetError: 65051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.077695] LNet: Removed LNI 192.168.204.33@tcp [ 1250.091363] LNet: Removed LNI 192.168.204.33@tcp1 [ 1250.562429] Key type .llcrypt unregistered [ 1250.564670] Key type ._llcrypt unregistered [ 1265.835587] Key type ._llcrypt registered [ 1265.837590] Key type .llcrypt registered [ 1267.277581] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:05:32 (1787580332) [ 1268.795672] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1270.498782] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:05:36 (1787580336) [ 1271.580419] Key type .llcrypt unregistered [ 1271.587772] Key type ._llcrypt unregistered [ 1277.884607] Key type ._llcrypt registered [ 1277.886717] Key type .llcrypt registered [ 1278.152151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.168424] alg: No test for adler32 (adler32-zlib) [ 1279.074601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1279.102568] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1286.423267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1287.191186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1287.231843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1287.291365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1287.313790] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1287.328378] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1287.610900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1287.666358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1287.714765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1287.777850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1287.822385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1287.891374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1287.956111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.003947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.026335] LNet: 66600:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1288.072158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1288.133220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1288.199756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1288.295403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.344565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1288.355536] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1288.405899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1288.783388] LNet: 66643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.789234] LNetError: 66643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.831970] LNet: Removed LNI 192.168.204.33@tcp [ 1290.360746] Key type .llcrypt unregistered [ 1290.364291] Key type ._llcrypt unregistered [ 1309.614904] Key type ._llcrypt registered [ 1309.618821] Key type .llcrypt registered [ 1309.804800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1309.822799] alg: No test for adler32 (adler32-zlib) [ 1310.712267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1310.747580] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1318.080521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1328.042158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1328.127736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1328.165660] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1329.457420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1329.500713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e network_timeout -r 1 [ 1329.559429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1329.601795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e network_timeout -r 1 [ 1329.669768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e network_timeout -r 1 [ 1329.717250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e network_timeout -r 1 [ 1329.740884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e network_timeout -r 1 [ 1329.782854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e network_timeout -r 1 [ 1329.805793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1329.839621] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1330.120403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1330.144488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1330.166677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1330.201909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1330.210728] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1330.216894] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1330.229859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1330.257320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1330.282935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1330.306621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1330.318712] LNet: 67528:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1330.331664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1330.350750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1330.371828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1330.398128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1330.427444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1330.468940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1335.355799] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1335.369674] LNet: Skipped 2 previous similar messages [ 1335.669216] LNet: 67606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.694193] LNetError: 67606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.737371] LNet: Removed LNI 192.168.204.33@tcp [ 1336.749110] LNetError: 67606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.774468] LNet: Removed LNI 192.168.204.33@tcp1 [ 1337.327809] Key type .llcrypt unregistered [ 1337.329925] Key type ._llcrypt unregistered [ 1353.599936] Key type ._llcrypt registered [ 1353.601863] Key type .llcrypt registered [ 1354.926221] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:07:00 (1787580420) [ 1355.181719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1355.192486] alg: No test for adler32 (adler32-zlib) [ 1356.113229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.196688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1356.212211] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1356.809859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1356.828461] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1357.735716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1357.800487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1357.849067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1357.902905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1357.965916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1358.009894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1358.090992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1358.123874] LNetError: 68706:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1358.243786] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1358.243824] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1360.289102] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1360.571338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1364.394776] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1365.149643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1365.190306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1365.279045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.318953] LNet: 68813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.325284] LNetError: 68813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.368385] LNet: Removed LNI 192.168.204.33@tcp [ 1366.372125] LNetError: 68813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1366.398977] LNet: Removed LNI 192.168.204.33@tcp1 [ 1366.422623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1366.469731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1366.492034] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1367.457595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1367.487462] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1368.659532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1368.733415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1368.859699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1368.910583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1368.989996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1369.052205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1369.102477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1369.154292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1369.178490] LNetError: 69086:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1369.505844] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1371.850679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1375.648914] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1375.655064] LNet: Skipped 1 previous similar message [ 1380.015916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1380.045332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1380.088697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1387.882979] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:07:33 (1787580453) [ 1388.202226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.222516] LNet: 69795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.228387] LNetError: 69795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.234949] LNet: Removed LNI 192.168.204.33@tcp [ 1388.253279] LNet: Removed LNI 192.168.204.33@tcp1 [ 1388.275957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.345451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1388.399307] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1389.257598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1389.276713] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1390.304204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1390.375956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1390.407990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1390.444565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1390.489665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1390.530913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1390.564470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 0 [ 1390.726710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1391.392991] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.33@tcp [ 1391.393060] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1391.412854] LNet: Skipped 1 previous similar message [ 1406.207744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1406.407284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1406.462353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1406.502327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 500 [ 1406.656554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.317509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1413.400194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1413.420965] LNet: 70287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.425871] LNetError: 70287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.437925] LNetError: 70287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1413.445618] LNet: Removed LNI 192.168.204.33@tcp [ 1413.466141] LNet: Removed LNI 192.168.204.33@tcp1 [ 1413.531266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.582179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1413.637428] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1414.599552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1414.618624] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1415.505346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1415.549453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1415.632944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1415.675128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1415.720254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1415.761687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1415.809450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1415.836733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.33@tcp --health 0 [ 1416.673471] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1416.677151] LNet: Skipped 3 previous similar messages [ 1417.136632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1427.378200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1427.403143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1427.426495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1433.422507] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:08:19 (1787580499) [ 1433.835256] LNet: 71333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.840497] LNetError: 71333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.845433] LNetError: 71333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1433.850766] LNet: Removed LNI 192.168.204.33@tcp [ 1433.874329] LNet: Removed LNI 192.168.204.33@tcp1 [ 1434.576650] Key type .llcrypt unregistered [ 1434.578669] Key type ._llcrypt unregistered [ 1441.037687] Key type ._llcrypt registered [ 1441.040618] Key type .llcrypt registered [ 1441.297318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1441.320459] alg: No test for adler32 (adler32-zlib) [ 1442.218266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1442.241076] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1451.110479] Lustre: DEBUG MARKER: Initial discovery [ 1451.181041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1453.572219] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1453.829608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1453.870825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1453.903288] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1453.932546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1453.972217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.133@tcp -r 1 -m GET -l 3 [ 1454.009229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.133@tcp -r 1 -m GET -e local_error [ 1454.056290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.204.133@tcp -r 1 -m PUT -l 6 [ 1454.565489] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1455.385088] Lustre: DEBUG MARKER: Force oleg433-client.virtnet to discover 192.168.204.133@tcp (in background) [ 1455.638170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.133@tcp [ 1456.960975] Lustre: DEBUG MARKER: Force oleg433-server to discover 192.168.204.33@tcp [ 1458.720623] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1459.521488] Lustre: DEBUG MARKER: Wait for 72102 [ 1461.033485] Lustre: DEBUG MARKER: Finished wait on 72102 [ 1461.213868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1461.250760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1461.252558] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1461.640912] LNet: 72554:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.648460] LNetError: 72554:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.668900] LNet: Removed LNI 192.168.204.33@tcp [ 1462.103723] Key type .llcrypt unregistered [ 1462.107066] Key type ._llcrypt unregistered [ 1479.489785] Key type ._llcrypt registered [ 1479.492483] Key type .llcrypt registered [ 1480.687290] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:09:06 (1787580546) [ 1481.177892] Key type .llcrypt unregistered [ 1481.183295] Key type ._llcrypt unregistered [ 1487.080348] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1487.246834] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1487.258514] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1487.271902] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1489.795785] Key type ._llcrypt registered [ 1489.801075] Key type .llcrypt registered [ 1490.028177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1490.053178] alg: No test for adler32 (adler32-zlib) [ 1491.098607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.163741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.211914] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1491.942368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1491.978968] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1500.455306] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:09:26 (1787580566) [ 1500.621882] LNet: 74656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.629633] LNetError: 74656:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1500.638121] LNet: Removed LNI 10.1.2.3@tcp [ 1500.653656] LNet: Removed LNI 192.168.204.33@tcp [ 1501.374043] Key type .llcrypt unregistered [ 1501.379355] Key type ._llcrypt unregistered [ 1508.654891] Key type ._llcrypt registered [ 1508.657694] Key type .llcrypt registered [ 1508.883044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1508.903109] alg: No test for adler32 (adler32-zlib) [ 1509.802502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.869913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1509.920250] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1510.779501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1510.822085] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1512.115082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.33@tcp [ 1513.184402] LNetError: Unexpected error -22 connecting to 192.168.204.33@tcp at host 192.168.204.33:988 [ 1518.452932] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:09:44 (1787580584) [ 1518.778730] LNet: 75796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1518.788597] LNetError: 75796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1518.797192] LNet: Removed LNI 10.1.2.3@tcp [ 1518.805928] LNet: Removed LNI 192.168.204.33@tcp [ 1519.593530] Key type .llcrypt unregistered [ 1519.597298] Key type ._llcrypt unregistered [ 1527.199414] Key type ._llcrypt registered [ 1527.202655] Key type .llcrypt registered [ 1527.439987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.467156] alg: No test for adler32 (adler32-zlib) [ 1528.424633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.499951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1528.543122] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1529.206941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1529.227668] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 1529.948234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.33@tcp1 --nid 192.168.204.33@tcp2 [ 1534.008939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp1 192.168.204.33@tcp2 [ 1534.031953] LNetError: 76636:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.33@tcp1 to 192.168.204.33@tcp2 - Route table may be misconfigured [ 1534.054710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp2 192.168.204.33@tcp1 [ 1539.932510] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:10:05 (1787580605) [ 1540.009836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.053566] LNet: 77204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1540.062879] LNetError: 77204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.070891] LNet: Removed LNI 192.168.204.33@tcp1 [ 1541.096661] LNet: Removed LNI 192.168.204.33@tcp2 [ 1541.128855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1541.208412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1541.241607] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1541.835724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1541.855685] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1542.644321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 1542.673859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.33@tcp [ 1542.703828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp1 192.168.204.33@tcp1 [ 1542.744252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.33@tcp -d 192.168.204.33@tcp -e network_timeout [ 1542.776508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -e network_timeout [ 1542.817980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -e network_timeout [ 1542.871093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -e network_timeout [ 1542.937769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1542.972476] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1542.999211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1543.128135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1543.200347] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1543.246538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1548.950914] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:10:14 (1787580614) [ 1549.199383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.230913] LNet: 78071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.250653] LNetError: 78071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.254710] LNetError: 78071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1549.261308] LNet: Removed LNI 192.168.204.33@tcp [ 1550.316031] LNet: Removed LNI 192.168.204.33@tcp1 [ 1550.362260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.470060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1550.793767] LNet: 78112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.121164] Key type .llcrypt unregistered [ 1551.123427] Key type ._llcrypt unregistered [ 1561.497374] Key type ._llcrypt registered [ 1561.502345] Key type .llcrypt registered [ 1562.573260] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:10:28 (1787580628) [ 1563.026602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1563.038412] alg: No test for adler32 (adler32-zlib) [ 1563.915191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.426478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1567.097738] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:10:32 (1787580632) [ 1567.245835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.276866] LNet: 79210:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.296957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.358848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.388425] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1568.108835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1568.127568] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1568.777530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1568.817817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp1 [ 1568.849919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp1 [ 1574.882727] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:10:40 (1787580640) [ 1576.831334] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg433-server [ 1578.594370] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:10:44 (1787580644) [ 1580.176700] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg433-server [ 1581.803303] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:10:47 (1787580647) [ 1583.533815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg433-server [ 1585.387721] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:10:51 (1787580651) [ 1587.066916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg433-server [ 1588.856393] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:10:54 (1787580654) [ 1590.603665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg433-server [ 1592.145381] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:10:58 (1787580658) [ 1593.937417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg433-server [ 1595.866704] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:11:01 (1787580661) [ 1597.807090] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg433-server [ 1599.296488] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:11:05 (1787580665) [ 1601.059426] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg433-server [ 1602.884768] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:11:08 (1787580668) [ 1604.587441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg433-server [ 1606.011087] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:11:12 (1787580672) [ 1606.135903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.171835] LNet: 83083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.189720] LNetError: 83083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.205447] LNet: Removed LNI 192.168.204.33@tcp [ 1607.274577] LNet: Removed LNI 192.168.204.33@tcp1 [ 1607.319469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.409280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1607.442353] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1608.143354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1608.216962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1608.409994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.434823] LNet: 83236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.441335] LNetError: 83236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.447293] LNetError: 83236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1608.458201] LNet: Removed LNI 192.168.204.33@tcp [ 1608.491990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.545875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1608.588177] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1609.377521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1609.532154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.545867] LNetError: 83381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.559933] LNet: Removed LNI 192.168.204.33@tcp [ 1609.587959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.635709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.680056] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1610.404843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1610.467350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1610.631712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1610.655318] LNet: 83528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1610.660375] LNet: 83528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1611.692095] LNet: Removed LNI 192.168.204.33@tcp [ 1611.715171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.782901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1611.814963] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1612.385903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1612.592563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.624475] LNetError: 83673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1612.636230] LNetError: 83673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1612.657224] LNet: Removed LNI 192.168.204.33@tcp [ 1612.697845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.742154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1612.758969] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1613.485348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1613.559248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1613.819935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.885812] LNet: Removed LNI 192.168.204.33@tcp [ 1614.903884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.943375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1614.975690] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1615.698868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1617.031210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.072230] LNet: 83975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.079070] LNet: 83975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1617.085340] LNetError: 83975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.091143] LNetError: 83975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1618.152838] LNet: Removed LNI 192.168.204.33@tcp [ 1618.185386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.241510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.269319] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1618.924726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1619.026708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1619.337250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1620.392249] LNet: Removed LNI 192.168.204.33@tcp [ 1620.435924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.530753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1620.569661] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1621.429654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1621.697470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.760556] LNet: Removed LNI 192.168.204.33@tcp [ 1622.799049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.881987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1622.910939] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1623.747549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1623.893445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1624.230662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1625.320450] LNet: Removed LNI 192.168.204.33@tcp [ 1625.368216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1625.464207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1625.509050] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1626.348171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1626.655468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.688115] LNet: 84560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.696626] LNet: 84560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1626.706482] LNetError: 84560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.715784] LNetError: 84560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1626.741929] LNet: Removed LNI 192.168.204.33@tcp [ 1626.780402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.853151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.892056] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1627.665414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1627.792945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1628.166709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.229360] LNet: Removed LNI 192.168.204.33@tcp [ 1629.270170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1629.347783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1629.387872] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1630.391497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1631.890327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.000475] LNet: Removed LNI 192.168.204.33@tcp [ 1633.062322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.137329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1633.217742] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1633.974769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1634.059491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1634.108843] LNet: Removed LNI 192.168.204.33@tcp [ 1634.133535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1634.213855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1634.238345] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1634.925412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1634.976399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.005539] LNet: Removed LNI 192.168.204.33@tcp [ 1635.019541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.079870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1635.164840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.215863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.253210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1635.330933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.380623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.442472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1635.521388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.567363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.627836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1635.658978] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1636.332727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.373623] LNet: Removed LNI 192.168.204.33@tcp [ 1636.394827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.492437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1644.171729] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:11:49 (1787580709) [ 1644.430774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.502049] LNet: 85883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1644.512512] LNet: 85883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1644.543843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.618483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1644.646099] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1646.553701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1646.569926] LNetError: 86021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.577651] LNetError: 86021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1646.593639] LNet: Removed LNI 192.168.204.33@tcp231 [ 1646.619203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.727851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1646.753530] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1646.761808] LNet: 86029:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1647.146778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1647.174404] LNet: Removed LNI 192.168.204.33@tcp231 [ 1647.192991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1647.221770] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1648.517913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.547090] LNet: Removed LNI 192.168.204.33@tcp231 [ 1648.568691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.624586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1648.642481] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/0] [ 1648.647353] LNet: 86181:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1649.045980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1649.095284] LNet: Removed LNI 192.168.204.33@tcp231 [ 1649.115232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1649.143856] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/0] [ 1650.057835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.079360] LNet: Removed LNI 192.168.204.33@tcp231 [ 1650.096595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.159251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1650.182285] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/60] [ 1650.188954] LNet: 86333:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1650.434575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1650.462991] LNet: Removed LNI 192.168.204.33@tcp231 [ 1650.486622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1650.508648] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/60] [ 1651.791468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1651.819931] LNet: Removed LNI 192.168.204.33@tcp231 [ 1651.845927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1651.946995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1651.966491] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1652.422531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1652.471738] LNet: Removed LNI 192.168.204.33@tcp231 [ 1652.497084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1652.530245] LNet: Added LNI 192.168.204.33@tcp231 [8/256/0/180] [ 1659.156909] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:12:05 (1787580725) [ 1659.205239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.225352] LNet: Removed LNI 192.168.204.33@tcp231 [ 1659.241716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1659.311637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1659.335738] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1660.014745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1660.054531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1660.085309] LNet: Removed LNI 192.168.204.33@tcp232 [ 1660.105305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1660.136066] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1660.932894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1660.964840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1661.001416] LNet: Removed LNI 192.168.204.33@tcp232 [ 1661.033793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1661.068359] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1661.733470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1661.763158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1661.802615] LNet: Removed LNI 192.168.204.33@tcp232 [ 1661.823151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1661.844302] LNet: Added LNI 192.168.204.33@tcp232 [8/256/0/180] [ 1662.632133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1662.678824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1662.716263] LNet: Removed LNI 192.168.204.33@tcp232 [ 1662.736990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1668.492581] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:12:14 (1787580734) [ 1669.450108] Key type .llcrypt unregistered [ 1669.451851] Key type ._llcrypt unregistered [ 1674.579175] Key type ._llcrypt registered [ 1674.582834] Key type .llcrypt registered [ 1674.789958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.814762] alg: No test for adler32 (adler32-zlib) [ 1675.708489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1675.729168] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1682.258144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1690.969758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.998931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1691.017762] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1692.137731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_interrupt -r 2 [ 1692.176067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 2 [ 1692.214657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_interrupt -r 2 [ 1692.273155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 2 [ 1692.323243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_interrupt -r 2 [ 1692.377241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_interrupt -r 2 [ 1692.431056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_interrupt -r 2 [ 1692.473947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_interrupt -r 2 [ 1692.513817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1692.553567] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1692.559993] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1692.791837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1692.826735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1692.860505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1692.896132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1692.938569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1692.972871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1692.998298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1693.048953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1693.066890] LNet: 88987:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1693.080785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1693.107607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1693.139652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1693.162699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1693.201500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1693.246945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1693.879792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_dropped -r 2 [ 1693.919906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 2 [ 1693.960541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_dropped -r 2 [ 1694.008885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 2 [ 1694.078114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_dropped -r 2 [ 1694.130445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_dropped -r 2 [ 1694.175869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_dropped -r 2 [ 1694.221089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_dropped -r 2 [ 1694.275983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1694.308700] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1694.586562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1694.623367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1694.659403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1694.700568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1694.733839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1694.767193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1694.817834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1694.859340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1694.881694] LNet: 89107:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1694.905880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1694.940542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1694.967238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1694.993666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.021319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.044668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1695.588141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_aborted -r 2 [ 1695.621631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 2 [ 1695.655670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_aborted -r 2 [ 1695.689393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 2 [ 1695.723682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_aborted -r 2 [ 1695.767796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_aborted -r 2 [ 1695.800590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_aborted -r 2 [ 1695.849841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_aborted -r 2 [ 1695.889296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1695.912142] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1696.146808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1696.192683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1696.235265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1696.280548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1696.312785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1696.337278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1696.369698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1696.406089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1696.426097] LNet: 89227:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1696.444801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1696.468874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1696.493234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1696.522777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1696.552234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1696.579989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1697.121739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_no_route -r 2 [ 1697.176728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 2 [ 1697.209480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_no_route -r 2 [ 1697.244590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 2 [ 1697.302477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_no_route -r 2 [ 1697.347967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_no_route -r 2 [ 1697.385579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_no_route -r 2 [ 1697.425083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_no_route -r 2 [ 1697.472984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1697.701735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1697.725276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1697.748874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1697.778579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1697.815451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1697.839216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1697.889850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1697.925614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1697.959431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1698.001622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1698.026762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1698.051732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1698.077742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1698.110256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1698.654564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e local_timeout -r 2 [ 1698.695354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 2 [ 1698.729615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e local_timeout -r 2 [ 1698.763722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 2 [ 1698.797445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e local_timeout -r 2 [ 1698.844683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e local_timeout -r 2 [ 1698.887513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e local_timeout -r 2 [ 1698.932342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e local_timeout -r 2 [ 1698.969643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1698.991669] LNet: There was an unexpected network error while writing to 192.168.204.133: rc = -22 [ 1698.996686] LNet: Skipped 1 previous similar message [ 1699.258054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1699.288716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1699.334187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1699.385822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1699.440565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1699.473474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1699.510443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1699.558757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1699.581417] LNet: 89468:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1699.586111] LNet: 89468:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1699.613568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1699.655333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1699.687459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1699.713236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1699.746922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1699.785208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1700.296857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp -m GET -e remote_dropped -r 2 [ 1700.329599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 2 [ 1700.365621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp -m GET -e remote_dropped -r 2 [ 1700.414061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 2 [ 1700.462567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp -m GET -e remote_dropped -r 2 [ 1700.497674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.133@tcp1 -m GET -e remote_dropped -r 2 [ 1700.539597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp -m GET -e remote_dropped -r 2 [ 1700.594160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.33@tcp1 -d 192.168.204.33@tcp1 -m GET -e remote_dropped -r 2 [ 1700.643535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.33@tcp 192.168.204.133@tcp [ 1700.678707] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 1700.916729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1700.945637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1700.979497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1701.010254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1701.056842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1701.082226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1701.108886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1701.140658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1701.168482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1701.188972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1701.211638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1701.237684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1701.266926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1701.307893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1706.921692] LNet: 89666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.933233] LNetError: 89666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.939839] LNet: Removed LNI 192.168.204.33@tcp [ 1706.963363] LNet: Removed LNI 192.168.204.33@tcp1 [ 1707.377241] Key type .llcrypt unregistered [ 1707.384749] Key type ._llcrypt unregistered [ 1724.185910] Key type ._llcrypt registered [ 1724.189864] Key type .llcrypt registered [ 1725.368927] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:13:11 (1787580791) [ 1725.836808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1725.852262] alg: No test for adler32 (adler32-zlib) [ 1726.788374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.852414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1726.876506] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1727.772590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1727.808461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.33@tcp [ 1727.829412] LNetError: 90645:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1727.840617] LNet: 1 local NIs in recovery (showing 1): 192.168.204.33@tcp [ 1727.919886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.33@tcp [ 1728.021347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1734.082793] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:13:20 (1787580800) [ 1734.407722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.440783] LNet: 91235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1734.450417] LNetError: 91235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.466206] LNet: Removed LNI 192.168.204.33@tcp [ 1734.519936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.632471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1734.679757] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1735.479590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1735.506880] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 1736.528569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.33@tcp [ 1736.557222] LNetError: 91502:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.33@tcp to recovery [ 1736.621942] LNet: There was an unexpected network error while writing to 192.168.204.33: rc = -22 [ 1737.838984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.33@tcp [ 1744.999948] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:13:31 (1787580811) [ 1746.906074] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg433-server [ 1748.773322] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:13:34 (1787580814) [ 1750.529145] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg433-server [ 1752.053902] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:13:38 (1787580818) [ 1753.954459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg433-server [ 1755.616709] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:13:41 (1787580821) [ 1755.876661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.899201] LNet: 93123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.907139] LNetError: 93123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.917885] LNet: Removed LNI 192.168.204.33@tcp [ 1755.939582] LNet: Removed LNI 192.168.204.33@tcp1 [ 1755.975411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.044446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.081652] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1756.801216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1756.878613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1756.935753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1757.049251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1757.083864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1757.146661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1762.479454] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:13:48 (1787580828) [ 1762.678408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.705144] LNet: 93854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.713160] LNetError: 93854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.721833] LNetError: 93854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1762.746318] LNet: Removed LNI 192.168.204.33@tcp [ 1762.771814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1762.827175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1762.852037] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1769.502229] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:13:55 (1787580835) [ 1769.585458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.615126] LNet: 94551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.622187] LNetError: 94551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1769.663099] LNet: Removed LNI 192.168.204.33@tcp [ 1769.693342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.781602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1769.807610] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1775.772570] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:14:01 (1787580841) [ 1777.253949] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1779.011357] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:14:04 (1787580844) [ 1779.385644] LNet: 95593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.389678] LNetError: 95593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.407823] LNet: Removed LNI 192.168.204.33@tcp [ 1780.064730] Key type .llcrypt unregistered [ 1780.067453] Key type ._llcrypt unregistered [ 1786.228925] Key type ._llcrypt registered [ 1786.230505] Key type .llcrypt registered [ 1786.491356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1786.509974] alg: No test for adler32 (adler32-zlib) [ 1787.442532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1787.478167] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1794.174191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 1798.611707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.133@tcp [ 1798.939554] LNet: 96019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.949499] LNetError: 96019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.963722] LNet: Removed LNI 192.168.204.33@tcp [ 1799.374427] Key type .llcrypt unregistered [ 1799.376738] Key type ._llcrypt unregistered [ 1809.524700] Key type ._llcrypt registered [ 1809.526848] Key type .llcrypt registered [ 1810.465843] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1811.673382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1812.954276] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:14:39 (1787580879) [ 1814.502160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1815.790459] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:14:41 (1787580881) [ 1817.185922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg433-server [ 1818.830504] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:14:44 (1787580884) [ 1820.688368] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg433-server [ 1822.136929] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:14:48 (1787580888) [ 1823.207373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.347525] alg: No test for adler32 (adler32-zlib) [ 1824.387297] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 1824.633354] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1826.288323] Key type lgssc registered [ 1827.403949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1844.376916] Key type lgssc unregistered [ 1844.584682] LNet: 99195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.593265] LNetError: 99195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.611604] LNet: Removed LNI 192.168.204.33@tcp [ 1845.267773] Key type .llcrypt unregistered [ 1845.271445] Key type ._llcrypt unregistered [ 1858.769024] Key type ._llcrypt registered [ 1858.772136] Key type .llcrypt registered [ 1859.955496] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:15:25 (1787580925) [ 1860.351459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1860.383587] alg: No test for adler32 (adler32-zlib) [ 1861.353942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1861.410409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1861.455140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1861.498130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1861.532806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1867.631801] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:15:33 (1787580933) [ 1867.808597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.843732] LNet: 100608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1867.881844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.968515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1868.011220] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1868.928305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.945729] LNet: 100756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.949764] LNetError: 100756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1868.974219] LNet: Removed LNI 192.168.204.33@tcp [ 1868.998804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.057690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1869.083126] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1869.582757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1869.600457] LNetError: 100784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.618808] LNet: Removed LNI 192.168.204.33@tcp [ 1869.640980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.742874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1869.776434] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1875.816368] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:15:41 (1787580941) [ 1875.887912] LNet: 101367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.899395] LNet: 101367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1875.907959] LNetError: 101367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.926291] LNet: Removed LNI 192.168.204.33@tcp [ 1876.610822] Key type .llcrypt unregistered [ 1876.614338] Key type ._llcrypt unregistered [ 1883.316308] Key type ._llcrypt registered [ 1883.324954] Key type .llcrypt registered [ 1883.582839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1883.600896] alg: No test for adler32 (adler32-zlib) [ 1884.511778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.576813] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1884.579938] LNet: *** cfs_fail_loc=e003, val=100*** [ 1885.333187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1885.374683] LNet: *** cfs_fail_loc=e003, val=100*** [ 1885.382281] LNet: Skipped 2 previous similar messages [ 1885.393237] Ping reply received with mapping: [ 1885.400862] found metadata reply NI[192.168.204.33@tcp] [ 1885.407270] 21 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1885.412832] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1885.419153] 5a 5a 5a 5a 5a 5a 5a 5a [ 1885.569310] LNet: 101777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.583875] LNetError: 101777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.611227] LNet: Removed LNI 192.168.204.33@tcp [ 1886.304552] Key type .llcrypt unregistered [ 1886.308856] Key type ._llcrypt unregistered [ 1897.245825] Key type ._llcrypt registered [ 1897.249142] Key type .llcrypt registered [ 1898.444612] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:16:04 (1787580964) [ 1899.078496] Key type .llcrypt unregistered [ 1899.084914] Key type ._llcrypt unregistered [ 1904.449348] Key type ._llcrypt registered [ 1904.451651] Key type .llcrypt registered [ 1904.643662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1904.663624] alg: No test for adler32 (adler32-zlib) [ 1905.819809] LNetError: 102835:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1906.341395] Key type .llcrypt unregistered [ 1906.347083] Key type ._llcrypt unregistered [ 1910.705915] Key type ._llcrypt registered [ 1910.710889] Key type .llcrypt registered [ 1911.958113] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:16:17 (1787580977) [ 1912.400551] Key type .llcrypt unregistered [ 1912.403430] Key type ._llcrypt unregistered [ 1917.501813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1917.511944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1920.075524] Key type ._llcrypt registered [ 1920.082487] Key type .llcrypt registered [ 1920.263294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1920.286062] alg: No test for adler32 (adler32-zlib) [ 1921.249728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.330443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1930.406467] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:16:36 (1787580996) [ 1930.544498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1930.565612] LNet: 104623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.570416] LNetError: 104623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1930.581992] LNet: Removed LNI 10.1.2.3@tcp [ 1930.609060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.378425] Key type .llcrypt unregistered [ 1931.380098] Key type ._llcrypt unregistered [ 1937.378970] Key type ._llcrypt registered [ 1937.381637] Key type .llcrypt registered [ 1937.583806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1937.602950] alg: No test for adler32 (adler32-zlib) [ 1938.529094] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1938.609624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1938.811971] LNet: 104916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.821026] LNetError: 104916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.837616] LNet: Removed LNI 192.168.204.33@tcp [ 1939.642807] Key type .llcrypt unregistered [ 1939.645921] Key type ._llcrypt unregistered [ 1951.537478] Key type ._llcrypt registered [ 1951.548815] Key type .llcrypt registered [ 1953.286807] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:16:58 (1787581018) [ 1954.315784] Key type .llcrypt unregistered [ 1954.321312] Key type ._llcrypt unregistered [ 1962.130697] Key type ._llcrypt registered [ 1962.133303] Key type .llcrypt registered [ 1962.295918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1962.313594] alg: No test for adler32 (adler32-zlib) [ 1969.789455] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:17:15 (1787581035) [ 1969.867866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.914264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.009825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.052375] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1970.938977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.988566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1971.054435] LNet: Removed LNI 192.168.204.33@tcp [ 1971.444942] LNet: 106673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1971.929641] Key type .llcrypt unregistered [ 1971.937324] Key type ._llcrypt unregistered [ 1982.481317] Key type ._llcrypt registered [ 1982.486824] Key type .llcrypt registered [ 1983.511246] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:17:29 (1787581049) [ 1983.982973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.003530] alg: No test for adler32 (adler32-zlib) [ 1984.970303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.028511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1985.050393] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 1985.690558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 1985.745986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.33@tcp [ 1985.776376] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1985.782150] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1985.793222] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1985, 0, 0 [ 1985.799158] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1985.802023] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1985.806148] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1985.809468] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1985, 0, 0 [ 1985.813649] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1985.816657] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1985.819591] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1985.822593] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1985, 0, 2035 [ 1985.827877] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1985.832402] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1985.836304] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1985.838169] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1985, 0, 2035 [ 1985.841591] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1991.807992] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:17:37 (1787581057) [ 1992.294955] LNet: 108214:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1992.303163] LNetError: 108214:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1992.318157] LNet: Removed LNI 192.168.204.33@tcp [ 1993.012428] Key type .llcrypt unregistered [ 1993.016607] Key type ._llcrypt unregistered [ 1999.236236] Key type ._llcrypt registered [ 1999.238919] Key type .llcrypt registered [ 1999.431366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1999.459944] alg: No test for adler32 (adler32-zlib) [ 2000.410867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2000.440996] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2008.094587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2017.755121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.811611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2017.840273] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 2018.656043] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2018.822555] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2018.834120] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2018.841912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2021.233903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2021.261546] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2032.042968] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:18:17 (1787581097) [ 2032.141961] LNet: 109714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2032.147617] LNetError: 109714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2032.163536] LNet: Removed LNI 192.168.204.33@tcp [ 2033.184179] LNet: Removed LNI 192.168.204.33@tcp1 [ 2033.186424] LNetError: 109714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2033.197656] LNetError: 109714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2033.220777] LNet: Removed LNI 10.1.2.3@tcp99 [ 2033.927337] Key type .llcrypt unregistered [ 2033.928659] Key type ._llcrypt unregistered [ 2040.339348] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2040.349401] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2042.829222] Key type ._llcrypt registered [ 2042.831400] Key type .llcrypt registered [ 2043.082899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2043.094427] alg: No test for adler32 (adler32-zlib) [ 2044.001755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2044.081546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2044.111951] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2044.855685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2044.879784] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2047.303428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2047.365980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2047.412124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2047.435316] LNetError: 110472:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2047.471758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2047.525687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2048.073572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2048.186791] LNet: 110492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.192851] LNetError: 110492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2048.198281] LNet: Removed LNI 10.1.2.3@tcp [ 2049.259074] LNet: Removed LNI 192.168.204.33@tcp [ 2050.063901] Key type .llcrypt unregistered [ 2050.070201] Key type ._llcrypt unregistered [ 2061.274232] Key type ._llcrypt registered [ 2061.276758] Key type .llcrypt registered [ 2062.627616] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:18:48 (1787581128) [ 2063.035286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2063.049420] alg: No test for adler32 (adler32-zlib) [ 2063.958238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.017659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.051840] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2064.721669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2064.793527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg433-client@tcp [ 2070.881591] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:18:56 (1787581136) [ 2070.955446] LNet: 112022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2070.968864] LNetError: 112022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2070.987502] LNet: Removed LNI 192.168.204.33@tcp [ 2071.696400] Key type .llcrypt unregistered [ 2071.697735] Key type ._llcrypt unregistered [ 2078.431606] Key type ._llcrypt registered [ 2078.435616] Key type .llcrypt registered [ 2078.617747] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2078.637747] alg: No test for adler32 (adler32-zlib) [ 2079.571327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2079.615989] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2079.643697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2085.949594] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:19:11 (1787581151) [ 2087.465903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2088.920311] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:19:14 (1787581154) [ 2090.516745] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2092.135038] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:19:18 (1787581158) [ 2092.230695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.255101] LNet: 113448:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2092.260930] LNetError: 113448:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2092.279690] LNet: Removed LNI 192.168.204.33@tcp [ 2092.300038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2092.401971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2092.421813] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2093.117869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2093.160860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2099.997266] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:19:25 (1787581165) [ 2100.176291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.204173] LNet: 114162:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2100.220492] LNetError: 114162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.261186] LNet: Removed LNI 192.168.204.33@tcp [ 2100.283955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.356695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2100.418530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2101.048733] Key type .llcrypt unregistered [ 2101.051994] Key type ._llcrypt unregistered [ 2111.616989] Key type ._llcrypt registered [ 2111.622475] Key type .llcrypt registered [ 2113.075413] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:19:38 (1787581178) [ 2113.654785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2113.677383] alg: No test for adler32 (adler32-zlib) [ 2114.587873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.642210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.674179] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2115.305221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2115.371604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2121.018729] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:19:47 (1787581187) [ 2121.178578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.205526] LNet: 115722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2121.221220] LNetError: 115722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.244819] LNet: Removed LNI 192.168.204.33@tcp [ 2121.268764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.330902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2121.364877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2126.851933] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:19:52 (1787581192) [ 2126.961085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2126.991745] LNet: 116303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.034332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.091298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2127.329212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2127.360612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2127.401178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2127.688996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2127.722310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2127.788544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.817628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.867587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2128.177558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.208718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.249769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2128.560902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.603980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2134.239156] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:20:00 (1787581200) [ 2134.348691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.393122] LNet: 116933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.398767] LNet: 116933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2134.422520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.478790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.500453] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2135.170749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.198783] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2135.945846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2136.013143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2136.231693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.247496] LNetError: 117218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.316343] LNet: Removed LNI 192.168.204.33@tcp [ 2137.323755] LNetError: 117218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.339375] LNet: Removed LNI 192.168.204.33@tcp2 [ 2137.378701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.465919] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2137.475021] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2137.481680] LNet: 117224:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2137.837674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2138.014141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.044993] LNet: 117261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.054514] LNet: 117261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2139.105026] LNet: Removed LNI 192.168.204.33@tcp [ 2139.120412] LNet: Removed LNI 192.168.204.33@tcp2 [ 2139.157852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.257944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.295726] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2139.840455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2139.867929] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2140.626617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2140.696967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2140.773426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2140.992198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.017945] LNetError: 117546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.025085] LNetError: 117546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2141.037169] LNet: Removed LNI 192.168.204.33@tcp [ 2142.056197] LNet: Removed LNI 192.168.204.33@tcp2 [ 2142.095616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.151663] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2142.156679] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2142.162254] LNet: 117552:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2142.537896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2142.787542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.806687] LNet: 117589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.812833] LNet: 117589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2142.830130] LNet: Removed LNI 192.168.204.33@tcp [ 2143.847958] LNet: Removed LNI 192.168.204.33@tcp2 [ 2143.862772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2143.903426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2143.941127] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2144.684650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2144.701902] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2145.462623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2145.531378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2145.754063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2145.781592] LNetError: 117869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2145.789772] LNetError: 117869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2145.795402] LNet: Removed LNI 192.168.204.33@tcp [ 2145.809084] LNet: Removed LNI 192.168.204.33@tcp2 [ 2145.834598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2145.913023] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2145.930853] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2145.946824] LNet: 117875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2146.439801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2146.657232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.744141] LNet: Removed LNI 192.168.204.33@tcp [ 2147.755785] LNet: Removed LNI 192.168.204.33@tcp2 [ 2147.790151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2147.844917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2147.869759] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2148.407774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2148.429803] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2149.137404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2149.191310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2149.258847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2149.465445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.497800] LNet: Removed LNI 192.168.204.33@tcp [ 2151.525984] LNet: Removed LNI 192.168.204.33@tcp2 [ 2151.564663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.622552] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2151.648197] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2151.654992] LNet: 118203:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2152.151888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2152.372697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.404740] LNet: 118240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.416892] LNet: 118240:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2152.433854] LNet: Removed LNI 192.168.204.33@tcp [ 2153.515605] LNet: Removed LNI 192.168.204.33@tcp2 [ 2153.571312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2153.705605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2153.767848] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2154.859132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2154.891681] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2155.733864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2155.776806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2155.928326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2156.665451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.710684] LNetError: 118537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2156.722054] LNetError: 118537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2157.728212] LNet: Removed LNI 192.168.204.33@tcp [ 2157.740483] LNet: Removed LNI 192.168.204.33@tcp2 [ 2157.797356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.885554] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2157.900060] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2157.908949] LNet: 118543:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2158.674145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2159.160418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.224139] LNet: Removed LNI 192.168.204.33@tcp [ 2160.246285] LNet: Removed LNI 192.168.204.33@tcp2 [ 2160.311583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.405835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.445350] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2161.463954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2161.489805] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2162.276251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2162.310156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2162.332889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2162.394968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2162.750256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2163.808531] LNet: Removed LNI 192.168.204.33@tcp [ 2164.906919] LNet: Removed LNI 192.168.204.33@tcp2 [ 2164.948882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.042716] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2165.049555] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2165.724419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2173.243221] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2175.154178] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:20:40 (1787581240) [ 2175.463558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.506489] LNet: 119664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2175.514720] LNet: 119664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2175.525713] LNetError: 119664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.530380] LNetError: 119664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2175.536535] LNet: Removed LNI 192.168.204.33@tcp [ 2176.552832] LNet: Removed LNI 192.168.204.33@tcp2 [ 2176.594215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2177.061438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2177.086911] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2177.884492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2177.906495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2179.044750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.064712] LNet: Removed LNI 10.1.2.3@tcp [ 2179.084357] LNet: Removed LNI 192.168.204.33@tcp [ 2179.108645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.220685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2179.261216] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2179.266434] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2179.819534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2179.858855] LNet: Removed LNI 192.168.204.33@tcp [ 2179.871139] LNet: Removed LNI 10.1.2.3@tcp [ 2186.357751] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:20:52 (1787581252) [ 2186.587793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2186.639808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2186.715949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2186.737963] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2187.464096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2187.499870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.538689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.590079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.637195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.704100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.752662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.795911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.824234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.891496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2187.972494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.33@tcp [ 2188.115857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2193.985183] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:21:00 (1787581260) [ 2194.169529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.231876] LNet: Removed LNI 192.168.204.33@tcp [ 2194.252393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.394306] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2194.585192] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2194.597125] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2194.624501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2197.009805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2197.067394] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2197.685575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2197.712535] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2199.800234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2199.847526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2199.925484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.33@tcp -m PUT -e local_timeout -r 1 [ 2199.982303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2200.043875] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2200.051754] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2200.403795] LNetError: 121343:0:(lib-move.c:4801:lnet_parse()) 192.168.204.33@tcp, src 192.168.204.33@tcp: Dropping PUT (error -108 looking up sender) [ 2202.144464] LNet: Removed LNI 10.1.2.3@tcp [ 2203.193440] LNet: Removed LNI 192.168.204.33@tcp [ 2203.621815] Key type .llcrypt unregistered [ 2203.629189] Key type ._llcrypt unregistered [ 2216.273715] Key type ._llcrypt registered [ 2216.277180] Key type .llcrypt registered [ 2217.559246] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:21:23 (1787581283) [ 2217.957507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2217.970319] alg: No test for adler32 (adler32-zlib) [ 2218.904648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.223576] LNet: 122660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2219.982464] Key type .llcrypt unregistered [ 2219.990667] Key type ._llcrypt unregistered [ 2226.547820] Key type ._llcrypt registered [ 2226.553234] Key type .llcrypt registered [ 2226.829819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2226.857205] alg: No test for adler32 (adler32-zlib) [ 2227.820805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2227.847315] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2236.140292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2236.421801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2237.015036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.133@tcp [ 2237.089677] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.133@tcp [ 2237.307137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2237.351708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2237.424490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2237.465348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2237.524773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2237.557933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2237.598400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2237.643271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2237.672778] LNet: 123114:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2237.704826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2237.752790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2237.804784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2237.832656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2237.862679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2237.923364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2238.546293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2238.780637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2238.810601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2238.834624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2238.878744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2238.913627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2238.953569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2238.984382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2239.017470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2239.038306] LNet: 123232:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.069255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2239.122575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2239.182661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2239.217864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2239.254718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2239.334259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2240.115112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.133@tcp [ 2240.419698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2240.468431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2240.510988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2240.562813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2240.609961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2240.642879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2240.704643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2240.735769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2240.763551] LNet: 123349:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2240.778965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2240.808958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2240.856614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2240.890142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2240.910665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2240.942545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2241.640940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2241.957664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2241.991551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2242.030062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2242.056204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.094663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.121788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.168111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.244889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.295983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2242.339469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2242.412161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2242.451994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2242.488739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2242.531958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2242.933515] LNet: 123516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2242.945151] LNetError: 123516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2242.977505] LNet: Removed LNI 192.168.204.33@tcp [ 2243.318755] Key type .llcrypt unregistered [ 2243.325889] Key type ._llcrypt unregistered [ 2259.813463] Key type ._llcrypt registered [ 2259.817749] Key type .llcrypt registered [ 2261.789806] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:22:06 (1787581326) [ 2262.311129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.322366] alg: No test for adler32 (adler32-zlib) [ 2263.200452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.566746] LNet: 124404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.250643] Key type .llcrypt unregistered [ 2264.254785] Key type ._llcrypt unregistered [ 2271.879661] Key type ._llcrypt registered [ 2271.882436] Key type .llcrypt registered [ 2272.101434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2272.116291] alg: No test for adler32 (adler32-zlib) [ 2273.148911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2273.185903] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2283.662766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.133@tcp [ 2293.308289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.352462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2293.392915] LNet: Added LNI 192.168.204.33@tcp1 [8/256/0/180] [ 2294.314641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2294.921335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.133@tcp [ 2295.109238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2295.134433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2295.155153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2295.176254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2295.205902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2295.227626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2295.252733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2295.283697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2295.301664] LNet: 125061:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2295.325413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2295.355529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2295.389918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2295.426878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2295.482876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2295.516682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2296.148849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2296.415927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2296.460594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2296.491449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2296.534304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2296.570705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2296.606148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2296.631505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2296.671939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2296.689953] LNet: 125179:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2296.712617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2296.739666] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.133@tcp, 192.168.204.133@tcp1 [ 2296.779926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2296.816030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2296.842346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2296.898928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2296.952744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2297.771281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.133@tcp [ 2298.169259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2298.220262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2298.263526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2298.298990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2298.334900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2298.404586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2298.454609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2298.512527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2298.540391] LNet: 125296:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2298.570187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2298.611384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2298.644228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2298.685738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2298.737631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2298.783991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2299.502529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2299.757608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2299.803263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2299.852060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2299.880788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2299.934835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2299.986692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2300.029715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2300.073614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2300.143609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2300.193056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2300.222704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2300.266717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2300.298899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2300.345365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2305.701837] LNet: 125498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2305.715241] LNetError: 125498:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2305.727972] LNet: Removed LNI 192.168.204.33@tcp [ 2305.762603] LNet: Removed LNI 192.168.204.33@tcp1 [ 2306.270592] Key type .llcrypt unregistered [ 2306.272477] Key type ._llcrypt unregistered [ 2322.785945] Key type ._llcrypt registered [ 2322.787817] Key type .llcrypt registered [ 2324.278861] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:23:09 (1787581389) [ 2324.801607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2324.819552] alg: No test for adler32 (adler32-zlib) [ 2325.937628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.005916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2326.038382] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2326.984747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2327.030922] LNet: Added LNI 192.168.204.33@tcp2 [8/256/0/180] [ 2327.885381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.33@tcp [ 2328.007957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2329.057049] LNet: Removed LNI 192.168.204.33@tcp [ 2336.233245] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:23:22 (1787581402) [ 2337.771421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2339.819499] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:23:25 (1787581405) [ 2347.609619] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:23:33 (1787581413) [ 2354.583264] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:23:40 (1787581420) [ 2361.911845] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:23:47 (1787581427) [ 2369.500321] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:23:55 (1787581435) [ 2376.945174] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:24:02 (1787581442) [ 2384.345913] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:24:10 (1787581450) [ 2391.262099] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:24:16 (1787581456) [ 2398.175457] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:24:24 (1787581464) [ 2404.028462] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:24:29 (1787581469) [ 2410.761969] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:24:36 (1787581476) [ 2418.412728] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:24:44 (1787581484) [ 2425.655575] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:24:51 (1787581491) [ 2434.183903] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:24:59 (1787581499) [ 2443.294574] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:25:08 (1787581508) [ 2451.618932] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:25:17 (1787581517) [ 2459.067950] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:25:24 (1787581524) [ 2465.676949] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:25:31 (1787581531) [ 2465.917500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2465.939746] LNet: 137194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2465.944403] LNetError: 137194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2465.956468] LNet: Removed LNI 192.168.204.33@tcp2 [ 2465.997316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2466.128799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2466.162032] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2467.377194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.394506] LNet: 137336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2467.400613] LNetError: 137336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.417294] LNet: Removed LNI 192.168.204.33@tcp [ 2467.447580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2467.508270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2467.548138] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2474.829736] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:25:40 (1787581540) [ 2483.766532] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:25:49 (1787581549) [ 2492.320712] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:25:58 (1787581558) [ 2499.034260] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:26:05 (1787581565) [ 2504.779960] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:26:10 (1787581570) [ 2512.562562] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:26:18 (1787581578) [ 2521.215803] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:26:26 (1787581586) [ 2529.039739] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:26:34 (1787581594) [ 2534.927595] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:26:40 (1787581600) [ 2542.029765] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:26:47 (1787581607) [ 2550.260701] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:26:55 (1787581615) [ 2557.653807] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:27:03 (1787581623) [ 2565.458892] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2328 sec ========== 10:27:11 (1787581631) [ 2565.781936] LNet: 144802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2565.790750] LNetError: 144802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2565.810275] LNet: Removed LNI 192.168.204.33@tcp [ 2566.875572] Key type .llcrypt unregistered [ 2566.880085] Key type ._llcrypt unregistered [ 2578.518916] Key type ._llcrypt registered [ 2578.521548] Key type .llcrypt registered [ 2578.695945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2578.709693] alg: No test for adler32 (adler32-zlib) [ 2579.728363] Lustre: Lustre: Build Version: 2.17.57_80_g35a8f2d [ 2580.061571] LNet: Added LNI 192.168.204.33@tcp [8/256/0/180] [ 2581.712187] Key type lgssc registered [ 2582.849734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.079112] Lustre: Mounted lustre-client - version 2.17.57_80_g35a8f2d [ 2654.124298] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2662.682779] Lustre: Unmounted lustre-client [ 2700.644174] Key type lgssc unregistered [ 2700.939324] LNet: 147767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2700.960813] LNetError: 147767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2700.987578] LNet: Removed LNI 192.168.204.33@tcp [ 2702.274552] Key type .llcrypt unregistered [ 2702.279133] Key type ._llcrypt unregistered