[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499672218 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002095] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.005283] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007012] pid_max: default: 32768 minimum: 301 [ 0.009130] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011033] SELinux: Initializing. [ 0.012081] *** VALIDATE selinux *** [ 0.020548] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024582] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027110] *** VALIDATE tmpfs *** [ 0.029279] *** VALIDATE proc *** [ 0.030000] *** VALIDATE cgroup *** [ 0.030000] *** VALIDATE cgroup2 *** [ 0.030273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033027] Spectre V2 : User space: Vulnerable [ 0.034008] Speculative Store Bypass: Vulnerable [ 0.037552] debug: unmapping init [mem 0xffffffff84459000-0xffffffff84460fff] [ 0.039148] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040625] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041020] ... version: 2 [ 0.042011] ... bit width: 48 [ 0.043008] ... generic registers: 4 [ 0.044008] ... value mask: 0000ffffffffffff [ 0.045009] ... max period: 00007fffffffffff [ 0.046010] ... fixed-purpose events: 3 [ 0.047008] ... event mask: 000000070000000f [ 0.049035] rcu: Hierarchical SRCU implementation. [ 0.051355] smp: Bringing up secondary CPUs ... [ 0.052533] x86: Booting SMP configuration: [ 0.053017] .... node #0, CPUs: #1 #2 #3 [ 0.071135] smp: Brought up 1 node, 4 CPUs [ 0.073011] smpboot: Max logical packages: 1 [ 0.074018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.117448] node 0 deferred pages initialised in 38ms [ 0.119219] devtmpfs: initialized [ 0.120342] x86/mm: Memory block size: 128MB [ 0.126207] gcov: version magic: 0x41383552 [ 0.129564] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.134093] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.135357] pinctrl core: initialized pinctrl subsystem [ 0.136283] [ 0.136867] ************************************************************* [ 0.137024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138019] ** ** [ 0.139019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.140016] ** ** [ 0.141015] ** This means that this kernel is built to expose internal ** [ 0.142011] ** IOMMU data structures, which may compromise security on ** [ 0.143011] ** your system. ** [ 0.144013] ** ** [ 0.145013] ** If you see this message and you are not debugging the ** [ 0.146012] ** kernel, report this immediately to your vendor! ** [ 0.147012] ** ** [ 0.148017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.149012] ************************************************************* [ 0.150847] NET: Registered protocol family 16 [ 0.151515] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.152059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.153048] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.155010] cpuidle: using governor menu [ 0.156833] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.159558] PCI: Using configuration type 1 for base access [ 0.161217] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.172817] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.173045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.177064] cryptd: max_cpu_qlen set to 1000 [ 0.186418] ACPI: Added _OSI(Module Device) [ 0.187034] ACPI: Added _OSI(Processor Device) [ 0.188015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.192704] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.197457] ACPI: Interpreter enabled [ 0.198091] ACPI: PM: (supports S0 S3 S4 S5) [ 0.199016] ACPI: Using IOAPIC for interrupt routing [ 0.200132] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.201495] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213558] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.219786] acpiphp: Slot [2] registered [ 0.220163] acpiphp: Slot [3] registered [ 0.221111] acpiphp: Slot [4] registered [ 0.222123] acpiphp: Slot [5] registered [ 0.223161] acpiphp: Slot [6] registered [ 0.224163] acpiphp: Slot [7] registered [ 0.225125] acpiphp: Slot [8] registered [ 0.226091] acpiphp: Slot [9] registered [ 0.227147] acpiphp: Slot [10] registered [ 0.228082] acpiphp: Slot [11] registered [ 0.229078] acpiphp: Slot [12] registered [ 0.230090] acpiphp: Slot [13] registered [ 0.231084] acpiphp: Slot [14] registered [ 0.232098] acpiphp: Slot [15] registered [ 0.233151] acpiphp: Slot [16] registered [ 0.234150] acpiphp: Slot [17] registered [ 0.235175] acpiphp: Slot [18] registered [ 0.236093] acpiphp: Slot [19] registered [ 0.237080] acpiphp: Slot [20] registered [ 0.238097] acpiphp: Slot [21] registered [ 0.239086] acpiphp: Slot [22] registered [ 0.240107] acpiphp: Slot [23] registered [ 0.241964] acpiphp: Slot [24] registered [ 0.242109] acpiphp: Slot [25] registered [ 0.243102] acpiphp: Slot [26] registered [ 0.244110] acpiphp: Slot [27] registered [ 0.245120] acpiphp: Slot [28] registered [ 0.246102] acpiphp: Slot [29] registered [ 0.248106] acpiphp: Slot [30] registered [ 0.249093] acpiphp: Slot [31] registered [ 0.250056] PCI host bridge to bus 0000:00 [ 0.251019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.252212] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.253029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.254026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.255022] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.256034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.257161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260203] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.265208] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.270486] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.273012] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.274022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.276019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.277427] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.278752] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.279042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.280748] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.282875] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.289689] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.291012] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.295083] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.298013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.301015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.308027] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.313608] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.316027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.320020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.327030] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.335529] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.336419] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.337838] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.338891] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.339252] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.345045] iommu: Default domain type: Passthrough [ 0.346000] SCSI subsystem initialized [ 0.346188] ACPI: bus type USB registered [ 0.348149] usbcore: registered new interface driver usbfs [ 0.350203] usbcore: registered new interface driver hub [ 0.353144] usbcore: registered new device driver usb [ 0.356235] pps_core: LinuxPPS API ver. 1 registered [ 0.358011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.362074] PTP clock support registered [ 0.365097] EDAC MC: Ver: 3.0.0 [ 0.366363] PCI: Using ACPI for IRQ routing [ 0.367000] NetLabel: Initializing [ 0.367000] NetLabel: domain hash size = 128 [ 0.369010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.372144] NetLabel: unlabeled traffic allowed by default [ 0.375162] vgaarb: loaded [ 0.377109] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.378010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.382312] clocksource: Switched to clocksource kvm-clock [ 0.543991] VFS: Disk quotas dquot_6.6.0 [ 0.547805] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.554862] *** VALIDATE ramfs *** [ 0.557240] *** VALIDATE hugetlbfs *** [ 0.562573] pnp: PnP ACPI init [ 0.566423] pnp: PnP ACPI: found 6 devices [ 0.588317] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.591467] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.593587] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.596074] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.598801] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.604213] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.608422] NET: Registered protocol family 2 [ 0.611917] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.619929] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.627212] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.634426] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.637699] TCP: Hash tables configured (established 65536 bind 65536) [ 0.640911] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.645412] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.647926] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.650874] NET: Registered protocol family 1 [ 0.654604] RPC: Registered named UNIX socket transport module. [ 0.656766] RPC: Registered udp transport module. [ 0.662256] RPC: Registered tcp transport module. [ 0.665448] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.670579] NET: Registered protocol family 44 [ 0.672152] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.676384] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.678193] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.684283] PCI: CLS 0 bytes, default 64 [ 0.685520] Unpacking initramfs... [ 2.996534] debug: unmapping init [mem 0xffff8fbc7cc64000-0xffff8fbc7ffcffff] [ 3.015598] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.020268] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.025994] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.992924] Initialise system trusted keyrings [ 3.998940] Key type blacklist registered [ 4.011720] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.039604] zbud: loaded [ 4.051058] *** VALIDATE nfs *** [ 4.055262] *** VALIDATE nfs4 *** [ 4.060047] pstore: using deflate compression [ 4.073555] Platform Keyring initialized [ 4.465424] NET: Registered protocol family 38 [ 4.472899] Key type asymmetric registered [ 4.489876] Asymmetric key parser 'x509' registered [ 4.506650] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.521963] io scheduler mq-deadline registered [ 4.529516] io scheduler kyber registered [ 4.537792] io scheduler bfq registered [ 4.547312] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.556583] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.569982] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.578315] ACPI: Power Button [PWRF] [ 4.966138] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 5.241923] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 5.437905] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.487769] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.530371] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.538625] Non-volatile memory driver v1.3 [ 5.540926] Linux agpgart interface v0.103 [ 5.624270] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 5.631700] vda: detected capacity change from 0 to 68526080 [ 5.673945] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.683645] vdb: detected capacity change from 0 to 1073741824 [ 5.699508] libphy: Fixed MDIO Bus: probed [ 5.717299] usbcore: registered new interface driver usbserial_generic [ 5.724721] usbserial: USB Serial support registered for generic [ 5.730483] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.753068] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.756967] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.766707] mousedev: PS/2 mouse device common for all mice [ 5.778989] rtc_cmos 00:05: RTC can wake from S4 [ 5.793330] rtc_cmos 00:05: registered as rtc0 [ 5.799908] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.814941] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.825369] intel_pstate: CPU model not supported [ 5.860620] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.863542] hid: raw HID events driver (C) Jiri Kosina [ 5.898461] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.909373] usbcore: registered new interface driver usbhid [ 5.917621] usbhid: USB HID core driver [ 5.921534] drop_monitor: Initializing network drop monitor service [ 5.930749] Initializing XFRM netlink socket [ 5.939382] NET: Registered protocol family 10 [ 5.953673] Segment Routing with IPv6 [ 5.957587] NET: Registered protocol family 17 [ 5.965516] mpls_gso: MPLS GSO support [ 5.990436] RAS: Correctable Errors collector initialized. [ 6.002262] AVX version of gcm_enc/dec engaged. [ 6.003904] AES CTR mode by8 optimization enabled [ 6.428903] sched_clock: Marking stable (6428870419, 0)->(7734054853, -1305184434) [ 6.446911] registered taskstats version 1 [ 6.448687] Loading compiled-in X.509 certificates [ 6.455948] zswap: loaded using pool lzo/zbud [ 6.597762] Key type big_key registered [ 6.665082] Key type encrypted registered [ 6.666446] ima: No TPM chip found, activating TPM-bypass! [ 6.675764] ima: Allocated hash algorithm: sha1 [ 6.682721] ima: No architecture policies found [ 6.692199] evm: Initialising EVM extended attributes: [ 6.698703] evm: security.selinux [ 6.704536] evm: security.ima [ 6.709935] evm: security.capability [ 6.714540] evm: HMAC attrs: 0x1 [ 6.724707] rtc_cmos 00:05: setting system clock to 2025-11-17 17:46:33 UTC (1763401593) [ 6.751838] debug: unmapping init [mem 0xffffffff85403000-0xffffffff855fffff] [ 6.769696] debug: unmapping init [mem 0xffffffff84182000-0xffffffff84458fff] [ 6.780185] Write protecting the kernel read-only data: 28672k [ 6.790559] debug: unmapping init [mem 0xffffffff82803000-0xffffffff829fffff] [ 6.801576] debug: unmapping init [mem 0xffffffff83114000-0xffffffff831fffff] [ 6.897903] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.927334] systemd[1]: Detected virtualization kvm. [ 6.940809] systemd[1]: Detected architecture x86-64. [ 6.945840] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.007695] systemd[1]: No hostname configured. [ 7.015919] systemd[1]: Set hostname to . [ 7.024253] random: systemd: uninitialized urandom read (16 bytes read) [ 7.033986] systemd[1]: Initializing machine ID from random generator. [ 7.475149] random: ln: uninitialized urandom read (6 bytes read) [ 8.102079] random: systemd: uninitialized urandom read (16 bytes read) [ 8.111544] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 8.131066] random: systemd: uninitialized urandom read (16 bytes read) [ 8.141572] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 8.151883] random: systemd: uninitialized urandom read (16 bytes read) [ 8.163074] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. Starting Setup Virtual Console... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.981811] device-mapper: uevent: version 1.0.3 [ 11.989598] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [[ 15.651341] random: fast init done  OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.856397] virtio_net virtio0 ens2: renamed from eth0 [ 17.159176] scsi host0: ata_piix [ 17.198198] scsi host1: ata_piix [ 17.199765] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 17.202323] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 22.257870] random: crng init done [ 22.259529] random: 5 urandom warning(s) missed due to ratelimiting [ 25.056720] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 28.351795] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 33.053766] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.818410] SELinux: Disabled at runtime. [ 34.021883] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 34.058147] systemd[1]: Detected virtualization kvm. [ 34.059571] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.330231] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.337153] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.347039] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.350787] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.355966] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.374634] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.393014] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 36.642910] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 38.938898] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ 39.140068] hrtimer: interrupt took 2153218 ns [ OK ] Started udev Kernel Device Manager. [ 40.342287] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 40.359253] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.460500] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.669416] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 46.935293] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit)[ 48.166781] NFS: Registering the id_resolver key type [ 48.168793] Key type id_resolver registered [ 48.170371] Key type id_legacy registered [ *] A start job is running for Configur…only root support (12s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg203-client login: [ 121.546708] libcfs: loading out-of-tree module taints kernel. [ 121.649599] Key type ._llcrypt registered [ 121.679933] Key type .llcrypt registered [ 122.142778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 122.171571] alg: No test for adler32 (adler32-zlib) [ 123.499250] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 124.187521] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 125.888530] Key type lgssc registered [ 127.202487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 232.173575] Lustre: Mounted lustre-client [ 236.478649] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 253.870441] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing check_logdir /tmp/testlogs/ [ 258.015450] Lustre: lustre-OST0000-osc-ffff8fbcd01d9800: disconnect after 23s idle [ 258.045588] Lustre: DEBUG MARKER: oleg203-client.virtnet: executing yml_node [ 262.155230] Lustre: DEBUG MARKER: Client: 2.16.61.42 [ 264.807609] Lustre: DEBUG MARKER: MDS: 2.16.61.42 [ 267.712400] Lustre: DEBUG MARKER: OSS: 2.16.61.42 [ 269.125958] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Nov 17 12:50:54 EST 2025 [ 282.158900] LustreError: 7918:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8fbcd01d9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 282.171738] LustreError: 7918:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 282.203132] Lustre: Unmounted lustre-client [ 315.792610] Key type lgssc unregistered [ 316.167740] LNet: 8365:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.175253] LNetError: 8365:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 316.198562] LNet: Removed LNI 192.168.202.3@tcp [ 317.015702] Key type .llcrypt unregistered [ 317.019821] Key type ._llcrypt unregistered [ 341.127271] Key type ._llcrypt registered [ 341.129024] Key type .llcrypt registered [ 341.297836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.310996] alg: No test for adler32 (adler32-zlib) [ 342.232477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 342.309734] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 342.328160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 346.255849] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 346.507821] LNet: 9238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.520295] LNetError: 9238:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.528836] LNet: Removed LNI 192.168.202.3@tcp [ 347.143462] Key type .llcrypt unregistered [ 347.146956] Key type ._llcrypt unregistered [ 353.069685] Key type ._llcrypt registered [ 353.071695] Key type .llcrypt registered [ 354.784860] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:52:19 (1763401939) [ 355.169757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 355.180275] alg: No test for adler32 (adler32-zlib) [ 356.121647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.228830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.244897] LNet: 9656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.264064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.315250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 356.330508] LNet: 9663:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 363.074835] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:52:28 (1763401948) [ 363.281583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.329424] LNet: 10241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.351791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.436721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 369.996709] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:52:35 (1763401955) [ 370.049252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.071788] LNet: 10819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.090076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.178139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 375.928107] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:52:41 (1763401961) [ 375.987989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.015785] LNet: 11406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.039505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.116649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 382.071957] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:52:47 (1763401967) [ 382.274859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.349074] LNet: 11995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.395198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.503869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 383.161351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 390.163525] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:52:55 (1763401975) [ 390.278323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.339173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.434360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 390.783770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 391.357168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 398.206311] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:53:03 (1763401983) [ 398.533455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.571787] LNet: 13183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 398.578846] LNet: 13183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 398.602347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.669238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 404.981973] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:53:10 (1763401990) [ 405.193222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.220528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.291922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 405.330541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 405.841913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 405.871684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 406.347963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 406.407238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 406.927496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 406.959753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 407.455895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 407.511110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 407.994249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 408.041895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.454240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 408.510589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 408.868587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 408.901199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.347971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 409.380655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 409.698618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 409.727688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.125226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 410.192947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 410.764388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 410.806202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 411.260621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 411.337714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 418.609203] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:53:23 (1763402003) [ 418.878535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.921571] LNet: 14449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.924433] LNet: 14449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 418.961488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.046128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 419.082904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 426.042923] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:53:31 (1763402011) [ 426.112956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.173953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.315443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.380249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 432.477627] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:53:38 (1763402018) [ 432.760048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.826639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.942201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 432.983379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 439.531414] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:53:44 (1763402024) [ 439.727647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.786733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.865192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 439.906887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 447.236507] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:53:52 (1763402032) [ 447.455443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.527777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.582394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 447.614059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 454.577779] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:53:59 (1763402039) [ 454.696291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.754991] LNet: 17375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.759750] LNet: 17375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 454.806831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.902617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 454.952372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 461.492833] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:54:06 (1763402046) [ 461.659958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.767034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.867574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.239937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 468.705311] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:54:14 (1763402054) [ 468.791462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.855551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.094146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.322776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.305399] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:54:20 (1763402060) [ 475.440304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.512359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.678955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 475.919950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 482.887852] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:54:28 (1763402068) [ 482.994147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.071982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.747714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.007255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 491.290510] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:54:36 (1763402076) [ 491.572987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.636288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.776863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.873166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 498.505812] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:54:44 (1763402084) [ 498.637140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.714570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.823740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.890939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 505.558918] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:54:50 (1763402090) [ 505.773697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.804381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.909740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 512.035463] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:54:57 (1763402097) [ 512.106084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.149290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.280519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 518.286600] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:55:03 (1763402103) [ 518.389049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.449144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.588426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 518.901994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 524.996282] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:55:10 (1763402110) [ 525.157357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.202696] LNet: 23400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.205196] LNet: 23400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 525.242510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.316697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 525.355353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 531.669025] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:55:17 (1763402117) [ 531.828232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.900877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.960877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 532.004870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 538.580969] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:55:24 (1763402124) [ 538.643678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.684543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.758439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 538.795641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 544.999084] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:55:30 (1763402130) [ 545.098300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.141392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.244691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 545.282113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 552.054266] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:55:37 (1763402137) [ 552.307748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 552.356443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 552.429930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 552.504219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 552.530638] LNetError: 25763:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 552.591381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 552.630668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 558.036959] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:55:43 (1763402143) [ 558.251738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.288800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.394625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8NECWEySFJ [ 558.473320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8NECWEySFJ [ 558.533353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.8NECWEySFJ [ 563.759938] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:55:49 (1763402149) [ 563.860035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.912632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.993819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 564.028770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 564.076390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 569.074395] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:55:54 (1763402154) [ 569.322322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.363452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.458517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 572.127206] LNet: Message forwarding will be enabled in 2s [ 576.223139] LNet: Message forwarding enabled [ 576.725299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 576.746869] LNet: Message forwarding will stop in 2s to 6s [ 581.187534] LNet: Message forwarding disabled [ 581.201248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 582.581327] LNet: Message forwarding will be enabled in 2s [ 586.340882] LNet: Message forwarding enabled [ 588.341858] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:56:13 (1763402173) [ 588.441519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 589.742212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.856975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 595.068793] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:56:20 (1763402180) [ 595.180267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.238651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.268484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 595.301477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 595.327733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 595.374570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 595.425535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 595.458634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 595.482371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 595.507316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 595.535886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 595.559079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 595.589723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 595.611303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 595.641207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 595.670301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 595.703575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 595.736058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 595.761861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 595.789489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 595.813840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 595.832795] LNetError: 28841:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 601.141880] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:56:26 (1763402186) [ 601.335659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 601.496453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 601.582529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 607.888834] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:56:33 (1763402193) [ 609.770782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg203-server [ 611.419294] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:56:36 (1763402196) [ 611.547664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.584988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.641793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.683385] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 613.142685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[4-8]@tcp [ 613.162252] LNet: 30556:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 613.225693] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.5@tcp is being used as a gateway but routing feature is not turned on [ 613.241241] LNetError: 30346:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.5@tcp has gone from up to down [ 614.249320] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 614.263882] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 615.273918] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.4@tcp is being used as a gateway but routing feature is not turned on [ 615.287865] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 617.320216] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.5@tcp is being used as a gateway but routing feature is not turned on [ 617.328081] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 618.569448] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:56:44 (1763402204) [ 620.247861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg203-server [ 621.420215] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.4@tcp is being used as a gateway but routing feature is not turned on [ 621.425562] LNetError: 30346:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 622.025787] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:56:47 (1763402207) [ 622.224264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.250382] LNetError: 31491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 622.279553] LNet: Removed LNI 192.168.202.3@tcp [ 622.308889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 622.377971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 622.404726] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 623.020599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[4-9/2]@tcp [ 623.045093] LNet: 31608:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 623.059985] LNet: 31608:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 623.090375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[4-9/2]@tcp [ 623.091791] LNetError: 31495:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.4@tcp has gone from up to down [ 623.122938] LNetError: 31495:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 629.146980] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:56:54 (1763402214) [ 629.564751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.590335] LNetError: 32201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.606900] LNet: Removed LNI 192.168.202.3@tcp [ 629.630170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.716145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 629.777280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.806717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.891947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 629.945601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.004831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.078502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 630.176769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.207599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.266443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.331259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.376758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.436919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 630.514972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.550112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.635854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.704916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.744616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.836219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 630.892895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.937710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.994951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.064847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.104660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.217454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 631.288464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.330739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.389765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 631.482964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.522911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.572069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 631.650358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.726165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.832647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 638.638724] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:57:04 (1763402224) [ 640.378242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg203-server [ 642.140223] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:57:07 (1763402227) [ 643.767851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg203-server [ 645.254965] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:57:10 (1763402230) [ 645.973647] Key type .llcrypt unregistered [ 645.976243] Key type ._llcrypt unregistered [ 650.976742] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 651.105477] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 651.112949] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 651.121452] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 653.658741] Key type ._llcrypt registered [ 653.660412] Key type .llcrypt registered [ 653.839819] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 653.862436] alg: No test for adler32 (adler32-zlib) [ 654.785502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.837736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.865370] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 655.453753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 655.469482] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 657.403203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 657.429902] LNet: Removed LNI 10.1.2.3@tcp [ 657.515904] LNet: 34320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.526501] LNetError: 34320:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.538521] LNet: Removed LNI 192.168.202.3@tcp [ 658.078619] Key type .llcrypt unregistered [ 658.084220] Key type ._llcrypt unregistered [ 667.858844] Key type ._llcrypt registered [ 667.862587] Key type .llcrypt registered [ 669.211209] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:57:34 (1763402254) [ 669.790952] Key type .llcrypt unregistered [ 669.793209] Key type ._llcrypt unregistered [ 674.556526] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 674.640158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 674.649939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 674.660812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 677.152389] Key type ._llcrypt registered [ 677.153961] Key type .llcrypt registered [ 677.315335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 677.330968] alg: No test for adler32 (adler32-zlib) [ 678.256639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 678.305163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 678.326788] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 678.967784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 679.007443] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 681.505691] LNet: 35803:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 681.519696] LNetError: 35803:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 681.526749] LNet: Removed LNI 192.168.202.3@tcp [ 681.540953] LNet: Removed LNI 10.1.2.3@tcp [ 682.126542] Key type .llcrypt unregistered [ 682.130997] Key type ._llcrypt unregistered [ 692.569356] Key type ._llcrypt registered [ 692.571147] Key type .llcrypt registered [ 693.843466] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:57:59 (1763402279) [ 694.401334] Key type .llcrypt unregistered [ 694.403311] Key type ._llcrypt unregistered [ 699.858634] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.065913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.072027] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.074746] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 702.711660] Key type ._llcrypt registered [ 702.713198] Key type .llcrypt registered [ 703.015560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.051580] alg: No test for adler32 (adler32-zlib) [ 703.964114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.004729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 704.029916] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 705.953122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 705.980202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.061800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 708.091034] LNet: Removed LNI 10.1.2.3@tcp [ 708.104397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 708.149830] LNet: Removed LNI 10.1.2.31@tcp [ 708.169259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.211418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 708.940532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 708.953833] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 709.962249] LNet: 37715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.968803] LNetError: 37715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 709.976041] LNet: Removed LNI 10.1.2.3@tcp [ 709.986483] LNet: Removed LNI 10.1.2.31@tcp [ 710.598529] Key type .llcrypt unregistered [ 710.601789] Key type ._llcrypt unregistered [ 719.436064] Key type ._llcrypt registered [ 719.437811] Key type .llcrypt registered [ 720.627264] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:58:26 (1763402306) [ 720.967493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 720.984214] alg: No test for adler32 (adler32-zlib) [ 721.998381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.066320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.084671] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 722.723587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.743467] LNet: 38665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.783147] LNetError: 38665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.812382] LNet: Removed LNI 192.168.202.3@tcp [ 722.830218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.957026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.3@tcp [ 722.991215] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 723.591751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.3@tcp [ 723.648778] LNet: Removed LNI 192.168.202.3@tcp [ 723.737694] LNet: 38800:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 724.456338] Key type .llcrypt unregistered [ 724.462860] Key type ._llcrypt unregistered [ 733.679356] Key type ._llcrypt registered [ 733.681170] Key type .llcrypt registered [ 734.688961] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:58:40 (1763402320) [ 736.317989] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg203-server [ 737.879053] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:58:43 (1763402323) [ 738.378585] Key type .llcrypt unregistered [ 738.381203] Key type ._llcrypt unregistered [ 742.941139] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 743.159017] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 743.167277] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 743.172133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 745.605544] Key type ._llcrypt registered [ 745.612246] Key type .llcrypt registered [ 745.771722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 745.799216] alg: No test for adler32 (adler32-zlib) [ 746.714387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.782818] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 747.405744] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 749.146722] LNet: 40629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.152788] LNetError: 40629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.156934] LNet: Removed LNI 192.168.202.3@tcp1 [ 749.168646] LNet: Removed LNI 10.1.2.3@tcp2 [ 749.680993] Key type .llcrypt unregistered [ 749.684108] Key type ._llcrypt unregistered [ 760.477171] Key type ._llcrypt registered [ 760.478902] Key type .llcrypt registered [ 761.826434] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:59:07 (1763402347) [ 762.248200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.278504] alg: No test for adler32 (adler32-zlib) [ 763.235916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.358343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 763.394904] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 764.418500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.442937] LNet: 41595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.458319] LNetError: 41595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.492269] LNet: Removed LNI 192.168.202.3@tcp [ 764.533863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.590820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 764.605360] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 772.003559] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:59:17 (1763402357) [ 772.307830] LNet: 42189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 772.319091] LNetError: 42189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 772.339136] LNet: Removed LNI 192.168.202.3@tcp [ 773.117553] Key type .llcrypt unregistered [ 773.122223] Key type ._llcrypt unregistered [ 778.382601] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 778.493758] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 778.501966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 778.507850] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 781.008678] Key type ._llcrypt registered [ 781.010494] Key type .llcrypt registered [ 781.185424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 781.210229] alg: No test for adler32 (adler32-zlib) [ 782.148262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.228851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 782.265858] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 783.122040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 783.168842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.311779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.330807] LNet: 42929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.335908] LNetError: 42929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.340832] LNet: Removed LNI 192.168.202.3@tcp [ 785.348765] LNet: Removed LNI 10.1.2.3@tcp [ 785.380763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.454395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 785.480546] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 785.483534] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.012065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 786.048406] LNet: 42956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.059433] LNetError: 42956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.075049] LNetError: 42956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 786.081106] LNet: Removed LNI 192.168.202.3@tcp [ 786.092629] LNet: Removed LNI 10.1.2.3@tcp [ 786.112374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 786.214277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 786.255427] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 786.260661] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 786.710434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 786.728529] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 787.725459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.764898] LNet: 43118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.775366] LNetError: 43118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.779630] LNetError: 43118:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 787.786668] LNet: Removed LNI 192.168.202.3@tcp [ 787.791777] LNet: Removed LNI 10.1.2.3@tcp [ 787.804222] LNet: Removed LNI 192.168.202.3@tcp2 [ 787.832689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.924917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 787.954531] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 787.959462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 787.966275] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 795.651116] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:59:40 (1763402380) [ 795.999458] LNet: 43717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.004812] LNetError: 43717:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.009399] LNetError: 43717:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 796.013714] LNet: Removed LNI 192.168.202.3@tcp [ 796.015671] LNet: Removed LNI 10.1.2.3@tcp [ 796.033302] LNet: Removed LNI 192.168.202.3@tcp2 [ 796.767693] Key type .llcrypt unregistered [ 796.771723] Key type ._llcrypt unregistered [ 802.125363] Key type ._llcrypt registered [ 802.127316] Key type .llcrypt registered [ 802.245753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.255135] alg: No test for adler32 (adler32-zlib) [ 803.164250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.206488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 803.226462] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 804.226564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.246742] LNet: 44107:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.262050] LNetError: 44107:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.304270] LNet: Removed LNI 192.168.202.3@tcp [ 804.331943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.396863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 804.442281] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 804.903991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.941672] LNet: 44137:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 804.947117] LNetError: 44137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.960963] LNet: Removed LNI 192.168.202.3@tcp [ 804.984398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.080349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 805.108936] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 805.563470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 805.609565] LNet: Removed LNI 192.168.202.3@tcp [ 805.620269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 805.706855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 805.735472] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 806.173164] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 806.300914] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 806.303500] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 806.312461] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 808.665570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.710278] LNet: 44243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.715146] LNet: 44243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 808.719945] LNetError: 44243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.736405] LNetError: 44243:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 808.762831] LNet: Removed LNI 192.168.202.3@tcp [ 808.820395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.879621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 808.911776] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 814.845584] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 13:00:00 (1763402400) [ 815.066241] LNet: 44840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.072045] LNetError: 44840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.087153] LNet: Removed LNI 192.168.202.3@tcp [ 815.708430] Key type .llcrypt unregistered [ 815.711694] Key type ._llcrypt unregistered [ 822.834745] Key type ._llcrypt registered [ 822.836379] Key type .llcrypt registered [ 823.225938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.256730] alg: No test for adler32 (adler32-zlib) [ 824.252959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.346895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.394859] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 825.483954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.520086] LNet: 45234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.534559] LNetError: 45234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.555111] LNet: Removed LNI 192.168.202.3@tcp [ 825.575641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.650729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 825.674199] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 833.111973] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 13:00:18 (1763402418) [ 833.324308] LNet: 45826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.333606] LNetError: 45826:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.347913] LNet: Removed LNI 192.168.202.3@tcp [ 834.040689] Key type .llcrypt unregistered [ 834.046599] Key type ._llcrypt unregistered [ 840.114685] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.262062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 840.272676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 840.280691] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.773749] Key type ._llcrypt registered [ 842.775242] Key type .llcrypt registered [ 842.985068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.004686] alg: No test for adler32 (adler32-zlib) [ 843.945993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.008815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 844.046893] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 844.902493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.915571] LNet: 46271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.919260] LNetError: 46271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.933754] LNet: Removed LNI 192.168.202.3@tcp [ 844.960332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.031242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 845.065841] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 852.141795] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 13:00:37 (1763402437) [ 852.251106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 852.297312] LNet: 46863:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 852.331542] LNetError: 46863:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 852.398709] LNet: Removed LNI 192.168.202.3@tcp [ 852.451768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.613388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 858.127963] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 13:00:43 (1763402443) [ 858.447781] LNet: 47451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.035086] Key type .llcrypt unregistered [ 859.036838] Key type ._llcrypt unregistered [ 864.593388] Key type ._llcrypt registered [ 864.594843] Key type .llcrypt registered [ 864.725812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.744041] alg: No test for adler32 (adler32-zlib) [ 865.653987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.699083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.725103] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 866.511604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.533288] LNet: 47845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.536617] LNetError: 47845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.566219] LNet: Removed LNI 192.168.202.3@tcp [ 866.595236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.666667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 866.693101] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 873.475441] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 13:00:58 (1763402458) [ 873.696970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.737778] LNet: 48438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.743829] LNetError: 48438:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.772153] LNet: Removed LNI 192.168.202.3@tcp [ 873.791474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.958124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 881.135473] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 13:01:06 (1763402466) [ 882.452217] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 884.119462] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 13:01:09 (1763402469) [ 885.372522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 887.130334] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 13:01:12 (1763402472) [ 888.528823] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 890.408954] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 13:01:15 (1763402475) [ 891.978837] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 893.553199] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 13:01:19 (1763402479) [ 895.124898] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 896.492669] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 13:01:22 (1763402482) [ 896.580809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 896.599776] LNet: 50498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.632386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.725985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 896.789962] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 897.481418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.3 [ 897.495823] LNet: Removed LNI 192.168.202.3@tcp [ 903.340285] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 13:01:28 (1763402488) [ 903.413108] LNet: 51201:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.419507] LNetError: 51201:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.437338] LNet: Removed LNI 192.168.202.3@tcp [ 904.053725] Key type .llcrypt unregistered [ 904.056924] Key type ._llcrypt unregistered [ 909.797636] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 910.051278] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 910.058377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 910.069163] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.701690] Key type ._llcrypt registered [ 912.703152] Key type .llcrypt registered [ 912.855254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.864312] alg: No test for adler32 (adler32-zlib) [ 914.140351] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.934266] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 13:01:45 (1763402505) [ 920.072458] LNet: 52104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.084042] LNetError: 52104:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.119790] LNet: Removed LNI 10.1.2.3@tcp [ 920.770287] Key type .llcrypt unregistered [ 920.773574] Key type ._llcrypt unregistered [ 926.670029] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 926.934907] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 926.944354] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 926.948477] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 929.747161] Key type ._llcrypt registered [ 929.748718] Key type .llcrypt registered [ 929.968340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 929.985476] alg: No test for adler32 (adler32-zlib) [ 931.020603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 937.829849] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 13:02:03 (1763402523) [ 937.992238] LNet: 52994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.000870] LNetError: 52994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 938.042259] LNet: Removed LNI 10.1.2.3@tcp [ 939.028939] Key type .llcrypt unregistered [ 939.030409] Key type ._llcrypt unregistered [ 944.188033] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 944.412758] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 944.417236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 944.427089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.989531] Key type ._llcrypt registered [ 946.991228] Key type .llcrypt registered [ 947.153849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.176310] alg: No test for adler32 (adler32-zlib) [ 948.168933] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.504394] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 13:02:19 (1763402539) [ 953.668897] LNet: 53888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.695605] LNetError: 53888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 953.716348] LNet: Removed LNI 10.1.2.3@tcp [ 954.275686] Key type .llcrypt unregistered [ 954.277479] Key type ._llcrypt unregistered [ 958.613382] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 958.802524] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 958.808910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 958.816446] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 961.420509] Key type ._llcrypt registered [ 961.422111] Key type .llcrypt registered [ 961.623283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.639395] alg: No test for adler32 (adler32-zlib) [ 962.589510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.683229] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.605421] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 13:02:35 (1763402555) [ 970.066556] LNet: 55114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.071649] LNetError: 55114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 970.082868] LNet: Removed LNI 10.1.2.3@tcp [ 970.587421] Key type .llcrypt unregistered [ 970.591845] Key type ._llcrypt unregistered [ 975.682692] Key type ._llcrypt registered [ 975.683868] Key type .llcrypt registered [ 975.888698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.903587] alg: No test for adler32 (adler32-zlib) [ 976.837960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 976.864094] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 983.545792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 984.371736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_interrupt -r 1 [ 984.412263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_interrupt -r 1 [ 984.446530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 984.463800] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 984.669386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 984.717580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 984.773875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 984.799462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 984.832819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 984.873475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 984.915589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 984.968214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 984.994758] LNet: 55547:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 985.017770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.059529] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 985.066461] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 985.074246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.113182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.149781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.184179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.216188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.238691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 985.740649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_dropped -r 1 [ 985.764646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_dropped -r 1 [ 985.794900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 986.005406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.046440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 986.089379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.137796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.180096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.221222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.254259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.290790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 986.306759] LNet: 55656:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 986.325668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 986.359370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 986.388892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 986.437330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 986.473683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 986.504825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 986.531184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 986.898495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_aborted -r 1 [ 986.939874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_aborted -r 1 [ 986.975776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 987.002166] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 987.005796] LNet: Skipped 1 previous similar message [ 987.166604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 987.203886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 987.237975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 987.277170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 987.322187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 987.364685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 987.410269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 987.436285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.450089] LNet: 55765:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 987.470460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.500149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.542924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.607560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.630288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.649266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.690262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 988.145518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_no_route -r 1 [ 988.180393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_no_route -r 1 [ 988.210869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 988.403621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.425642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 988.451177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.486572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.510643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.536280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.562575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.593715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.625956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.643962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.673247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.701173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.729741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.759817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.790509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 989.216350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_timeout -r 1 [ 989.240504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_timeout -r 1 [ 989.260579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 989.275686] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 989.278215] LNet: Skipped 2 previous similar messages [ 989.414918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.437543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 989.463193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.484408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.508266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.537569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.561656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.596504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.623542] LNet: 55984:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.633427] LNet: 55984:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 989.661163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.690965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.732057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.758173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.777757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.803490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.831429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 990.182587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_error -r 1 [ 990.209972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_error -r 1 [ 990.236863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 990.399120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 990.424966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 990.451082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.474496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.501682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.521525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.542704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.568199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.586234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.606430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.637669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.657417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.699738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.727299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.755411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 990.929952] LNet: 56133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.936807] LNetError: 56133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.977525] LNet: Removed LNI 192.168.202.3@tcp [ 992.238927] Key type .llcrypt unregistered [ 992.240929] Key type ._llcrypt unregistered [ 1004.007549] Key type ._llcrypt registered [ 1004.012415] Key type .llcrypt registered [ 1005.012576] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 13:03:10 (1763402590) [ 1005.868396] Key type .llcrypt unregistered [ 1005.869765] Key type ._llcrypt unregistered [ 1009.603660] Key type ._llcrypt registered [ 1009.608697] Key type .llcrypt registered [ 1009.740024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.767433] alg: No test for adler32 (adler32-zlib) [ 1010.640854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1010.665944] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1016.680135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1025.943450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.014530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1026.048503] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1027.518577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_interrupt -r 1 [ 1027.587804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 1 [ 1027.636938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_interrupt -r 1 [ 1027.668718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 1 [ 1027.700812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_interrupt -r 1 [ 1027.741385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 1 [ 1027.784641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_interrupt -r 1 [ 1027.825541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 1 [ 1027.877069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1027.893812] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1027.897464] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1028.116746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.145674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.169607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.199228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.233349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.280244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.316303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.347784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.378724] LNet: 57599:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.411715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.446024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.480553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.501428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.543133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.598050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.627552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.078390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_dropped -r 1 [ 1029.108220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 1 [ 1029.131510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_dropped -r 1 [ 1029.186393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 1 [ 1029.225793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_dropped -r 1 [ 1029.263587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 1 [ 1029.322757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_dropped -r 1 [ 1029.372924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 1 [ 1029.418697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1029.443516] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1029.447083] LNet: Skipped 2 previous similar messages [ 1029.628344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.654750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1029.689830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1029.720933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.744314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1029.775501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.845489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.871925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.905179] LNet: 57721:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1029.931030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.982624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.040682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.101561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.139473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.198567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.263884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1030.768915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_aborted -r 1 [ 1030.791519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 1 [ 1030.818481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_aborted -r 1 [ 1030.847800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 1 [ 1030.882330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_aborted -r 1 [ 1030.931266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 1 [ 1030.960854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_aborted -r 1 [ 1031.002068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 1 [ 1031.043362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1031.062466] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1031.069251] LNet: Skipped 2 previous similar messages [ 1031.331925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1031.350295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1031.383407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1031.423645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1031.469642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1031.495541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1031.548432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.579940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.598308] LNet: 57842:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1031.618556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.667640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.694541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.749233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.810666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.854353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.891288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1032.508988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_no_route -r 1 [ 1032.537291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 1 [ 1032.592752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_no_route -r 1 [ 1032.617796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 1 [ 1032.643308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_no_route -r 1 [ 1032.681285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 1 [ 1032.710382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_no_route -r 1 [ 1032.748655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 1 [ 1032.776959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1033.012712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.039562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.071566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.101442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.127562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.150420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1033.176542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1033.200559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1033.229186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1033.254457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1033.282979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1033.337291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1033.361902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1033.393807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1033.421895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1033.891235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_timeout -r 1 [ 1033.917989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 1 [ 1033.950930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_timeout -r 1 [ 1033.985234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 1 [ 1034.034124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_timeout -r 1 [ 1034.064975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 1 [ 1034.124387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_timeout -r 1 [ 1034.154584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 1 [ 1034.199679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1034.231195] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1034.234936] LNet: Skipped 5 previous similar messages [ 1034.475239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1034.512955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1034.548902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1034.590514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1034.613985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1034.634535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.658670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.689961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.706516] LNet: 58085:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1034.710586] LNet: 58085:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1034.724968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.759310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.789306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.845630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.889995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.918117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.938188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1035.360995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_error -r 1 [ 1035.404403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_error -r 1 [ 1035.452853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_error -r 1 [ 1035.479394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_error -r 1 [ 1035.517127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_error -r 1 [ 1035.576234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_error -r 1 [ 1035.623478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_error -r 1 [ 1035.663838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_error -r 1 [ 1035.722701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1035.996750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1036.021538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1036.044186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1036.073750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.096947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.134316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.171614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.209233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.234821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.261358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.282919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.319901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.360528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.390251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1036.423707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1040.970771] LNet: 58281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.974886] LNetError: 58281:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.980286] LNet: Removed LNI 192.168.202.3@tcp [ 1042.027236] LNet: Removed LNI 192.168.202.3@tcp1 [ 1042.428873] Key type .llcrypt unregistered [ 1042.432885] Key type ._llcrypt unregistered [ 1054.995291] Key type ._llcrypt registered [ 1054.998254] Key type .llcrypt registered [ 1056.348769] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 13:04:01 (1763402641) [ 1057.136343] Key type .llcrypt unregistered [ 1057.140620] Key type ._llcrypt unregistered [ 1062.176296] Key type ._llcrypt registered [ 1062.183624] Key type .llcrypt registered [ 1062.414370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1062.437085] alg: No test for adler32 (adler32-zlib) [ 1063.339803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1063.373951] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1070.583132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1071.431202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_dropped -r 1 [ 1071.460594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_dropped -r 1 [ 1071.499601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1071.529500] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1071.764604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.785765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1071.812354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.852319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.880451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.905398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.932737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.968610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.988314] LNet: 59526:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.003187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.027507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.056288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.086644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.117894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.148657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.181834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1072.645764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_error -r 1 [ 1072.679164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_error -r 1 [ 1072.709724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1072.734130] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1072.927358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.959590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1072.986655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.013315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.037941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.071258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.095486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.124121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.140653] LNet: 59636:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1073.160992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1073.191296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1073.216894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1073.246739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1073.278888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1073.310967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1073.336949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1073.819962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_timeout -r 1 [ 1073.844775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_timeout -r 1 [ 1073.887386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1073.905551] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1074.120779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1074.146328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1074.164109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1074.196268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1074.214742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1074.235220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1074.268632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1074.311698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1074.347632] LNet: 59745:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1074.379503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.409383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.447478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.471464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.499183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.532631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.587878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1074.869948] LNet: 59786:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.877387] LNetError: 59786:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.907825] LNet: Removed LNI 192.168.202.3@tcp [ 1075.204433] Key type .llcrypt unregistered [ 1075.205834] Key type ._llcrypt unregistered [ 1087.341380] Key type ._llcrypt registered [ 1087.342579] Key type .llcrypt registered [ 1088.611756] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 13:04:34 (1763402674) [ 1089.459404] Key type .llcrypt unregistered [ 1089.466149] Key type ._llcrypt unregistered [ 1095.704618] Key type ._llcrypt registered [ 1095.706239] Key type .llcrypt registered [ 1096.033366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1096.069909] alg: No test for adler32 (adler32-zlib) [ 1096.997380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1097.026522] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1102.395696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1109.065408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.103266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1109.118939] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1110.132937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_dropped -r 1 [ 1110.233413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 1 [ 1110.266561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_dropped -r 1 [ 1110.293726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 1 [ 1110.331548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_dropped -r 1 [ 1110.357872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 1 [ 1110.384269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_dropped -r 1 [ 1110.410389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 1 [ 1110.443286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1110.472201] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1110.474801] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 1110.645731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.667636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.692358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.719149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.750222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.779823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.802766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.827466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.842777] LNet: 61246:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.856224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.880561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.916468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.952451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.980213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.007198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.047689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.087429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.597887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_error -r 1 [ 1111.636778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_error -r 1 [ 1111.673266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_error -r 1 [ 1111.705439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_error -r 1 [ 1111.737794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_error -r 1 [ 1111.776381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_error -r 1 [ 1111.844208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_error -r 1 [ 1111.892956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_error -r 1 [ 1111.943882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1111.963667] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1111.967848] LNet: Skipped 2 previous similar messages [ 1112.161574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.192191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1112.227308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.268423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.296689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.319900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.349525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.382874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.403200] LNet: 61369:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.413542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.443603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.465553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.508269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.540640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.573245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.606019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.631809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.080232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_timeout -r 1 [ 1113.099987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_timeout -r 1 [ 1113.137224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_timeout -r 1 [ 1113.172851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_timeout -r 1 [ 1113.200440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_timeout -r 1 [ 1113.225398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_timeout -r 1 [ 1113.260596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_timeout -r 1 [ 1113.282908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_timeout -r 1 [ 1113.311817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.103@tcp [ 1113.327448] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1113.331804] LNet: Skipped 1 previous similar message [ 1113.495895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.528721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1113.547437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1113.578399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1113.611322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1113.664993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1113.696729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.724742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.741725] LNet: 61492:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.757916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.783590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.815465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.852780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.875061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.930202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.971716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1114.010699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.123438] LNet: 61569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.135821] LNetError: 61569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.199384] LNet: Removed LNI 192.168.202.3@tcp [ 1119.201328] LNetError: 61569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1119.224248] LNet: Removed LNI 192.168.202.3@tcp1 [ 1119.723234] Key type .llcrypt unregistered [ 1119.726476] Key type ._llcrypt unregistered [ 1129.890132] Key type ._llcrypt registered [ 1129.891342] Key type .llcrypt registered [ 1130.916335] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 13:05:16 (1763402716) [ 1131.394316] Key type .llcrypt unregistered [ 1131.396193] Key type ._llcrypt unregistered [ 1135.295588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1135.368252] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1135.373598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1135.379447] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1137.759197] Key type ._llcrypt registered [ 1137.761961] Key type .llcrypt registered [ 1137.917763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1137.928371] alg: No test for adler32 (adler32-zlib) [ 1138.881437] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1138.926842] LNet: 62665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1138.940796] LNetError: 62665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1138.955694] LNet: Removed LNI 192.168.202.3@tcp [ 1139.491605] Key type .llcrypt unregistered [ 1139.493208] Key type ._llcrypt unregistered [ 1143.956377] Key type ._llcrypt registered [ 1143.957850] Key type .llcrypt registered [ 1144.167547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.180615] alg: No test for adler32 (adler32-zlib) [ 1145.121443] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1145.131278] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1145.173437] LNet: 62934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1145.181767] LNetError: 62934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1145.186943] LNet: Removed LNI 192.168.202.3@tcp [ 1145.202493] LNet: Removed LNI 10.1.2.3@tcp1 [ 1145.816319] Key type .llcrypt unregistered [ 1145.828830] Key type ._llcrypt unregistered [ 1150.780129] Key type ._llcrypt registered [ 1150.782502] Key type .llcrypt registered [ 1150.948250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.966227] alg: No test for adler32 (adler32-zlib) [ 1151.893813] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1151.895882] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.927880] LNet: 63203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.931499] LNetError: 63203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1151.937559] LNet: Removed LNI 192.168.202.3@tcp [ 1151.944157] LNet: Removed LNI 10.1.2.3@tcp [ 1152.445121] Key type .llcrypt unregistered [ 1152.446732] Key type ._llcrypt unregistered [ 1156.113636] Key type ._llcrypt registered [ 1156.115360] Key type .llcrypt registered [ 1156.236246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1156.248868] alg: No test for adler32 (adler32-zlib) [ 1157.122831] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1157.125301] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1157.145829] LNet: 63474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1157.151946] LNetError: 63474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1157.155719] LNet: Removed LNI 192.168.202.3@tcp [ 1157.168662] LNet: Removed LNI 10.1.2.3@tcp [ 1157.554471] Key type .llcrypt unregistered [ 1157.556614] Key type ._llcrypt unregistered [ 1161.616103] Key type ._llcrypt registered [ 1161.617556] Key type .llcrypt registered [ 1161.730716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1161.741716] alg: No test for adler32 (adler32-zlib) [ 1162.619535] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1162.621706] LNetError: 63735:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1162.624772] LNetError: Error -100 starting up LNI tcp [ 1162.639948] LNet: Removed LNI 192.168.202.3@tcp [ 1163.082354] Key type .llcrypt unregistered [ 1163.083740] Key type ._llcrypt unregistered [ 1169.799250] Key type ._llcrypt registered [ 1169.802491] Key type .llcrypt registered [ 1170.675734] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 13:05:56 (1763402756) [ 1171.325274] Key type .llcrypt unregistered [ 1171.328115] Key type ._llcrypt unregistered [ 1175.155850] Key type ._llcrypt registered [ 1175.158784] Key type .llcrypt registered [ 1175.356492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1175.371361] alg: No test for adler32 (adler32-zlib) [ 1176.253524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1176.281862] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1181.765545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1182.343792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e network_timeout -r 1 [ 1182.370180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e network_timeout -r 1 [ 1182.403819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1182.427725] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1182.435536] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1182.577863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.607517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1182.642854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.686058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.710235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.746791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.768265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.798621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.817622] LNet: 64984:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.829303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.854955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.899838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.936237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.963979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.009361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.234495] LNet: 65022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1183.245477] LNetError: 65022:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1183.263439] LNet: Removed LNI 192.168.202.3@tcp [ 1183.580492] Key type .llcrypt unregistered [ 1183.582280] Key type ._llcrypt unregistered [ 1196.105524] Key type ._llcrypt registered [ 1196.106942] Key type .llcrypt registered [ 1196.262950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1196.277742] alg: No test for adler32 (adler32-zlib) [ 1197.124425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1197.147305] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1202.579036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1209.957223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1210.020930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1210.061668] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1211.629949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e network_timeout -r 1 [ 1211.660188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e network_timeout -r 1 [ 1211.698658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e network_timeout -r 1 [ 1211.726080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e network_timeout -r 1 [ 1211.753874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e network_timeout -r 1 [ 1211.781747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e network_timeout -r 1 [ 1211.809994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e network_timeout -r 1 [ 1211.844929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e network_timeout -r 1 [ 1211.910716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1211.926463] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1212.118244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.156587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.186685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.240279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.275961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.316350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.339353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.375489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.388185] LNet: 65894:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.404835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.426400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.462655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.497756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.529760] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1212.532726] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1212.533795] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 1212.541060] LNet: Skipped 1 previous similar message [ 1212.543492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.585563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.320929] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1216.641829] LNet: 65967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.655987] LNetError: 65967:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.673507] LNet: Removed LNI 192.168.202.3@tcp [ 1216.697508] LNet: Removed LNI 192.168.202.3@tcp1 [ 1217.134522] Key type .llcrypt unregistered [ 1217.136178] Key type ._llcrypt unregistered [ 1230.938847] Key type ._llcrypt registered [ 1230.940190] Key type .llcrypt registered [ 1232.064347] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 13:06:57 (1763402817) [ 1232.432761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.444241] alg: No test for adler32 (adler32-zlib) [ 1233.357354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1233.398137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1233.417961] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1234.078278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1234.094566] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1234.901951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1234.972981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1235.013703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1235.041385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1235.091679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1235.113662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1235.141980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.3@tcp [ 1235.157428] LNetError: 67064:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 1235.424903] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1235.426939] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1237.472552] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1238.586073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1241.568397] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1242.015753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1242.050955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.121310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1242.148284] LNet: 67171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1242.156599] LNetError: 67171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.167263] LNet: Removed LNI 192.168.202.3@tcp [ 1243.171289] LNetError: 67171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.189199] LNet: Removed LNI 192.168.202.3@tcp1 [ 1243.226983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1243.323705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1243.343355] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1244.067789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1244.088846] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1244.993823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1245.050714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.152326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1245.210325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1245.258181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1245.304656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1245.346277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1245.375807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.3@tcp [ 1245.390736] LNetError: 67444:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 1246.368634] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1248.856475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1252.511405] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1252.517880] LNet: Skipped 1 previous similar message [ 1257.628846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1257.653487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1257.677471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.688928] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 13:07:28 (1763402848) [ 1262.849423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1262.873345] LNet: 68171:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1262.878071] LNetError: 68171:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1262.882422] LNet: Removed LNI 192.168.202.3@tcp [ 1262.888770] LNet: Removed LNI 192.168.202.3@tcp1 [ 1262.902169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1262.946603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1262.986491] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1263.540401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1263.560597] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1264.223885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1264.289514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1264.309916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1264.340928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1264.376145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1264.404909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1264.439592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.3@tcp --health 0 [ 1264.518872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1264.991149] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.3@tcp [ 1266.015362] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1266.021302] LNet: Skipped 1 previous similar message [ 1280.548973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1280.654785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.676747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1280.698893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.3@tcp --health 500 [ 1280.764268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1287.347058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1287.406449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1287.420359] LNet: 68670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1287.426746] LNetError: 68670:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1287.430417] LNetError: 68670:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1287.434274] LNet: Removed LNI 192.168.202.3@tcp [ 1287.448095] LNet: Removed LNI 192.168.202.3@tcp1 [ 1287.490714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.558363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1287.580763] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1288.315199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1288.333954] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1289.121395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1289.164240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.236607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1289.268735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1289.316111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1289.356845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1289.409788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1289.444547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.3@tcp --health 0 [ 1290.602393] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1290.605669] LNet: Skipped 3 previous similar messages [ 1290.726907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1301.994228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1302.032796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1302.074263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1307.644692] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 13:08:13 (1763402893) [ 1308.207976] LNet: 69727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.213579] LNetError: 69727:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1308.221418] LNetError: 69727:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1308.239242] LNet: Removed LNI 192.168.202.3@tcp [ 1308.249056] LNet: Removed LNI 192.168.202.3@tcp1 [ 1308.767899] Key type .llcrypt unregistered [ 1308.769738] Key type ._llcrypt unregistered [ 1312.671797] Key type ._llcrypt registered [ 1312.674902] Key type .llcrypt registered [ 1312.954163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1312.970381] alg: No test for adler32 (adler32-zlib) [ 1313.867300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1313.887937] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1320.369570] Lustre: DEBUG MARKER: Initial discovery [ 1320.407192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.103@tcp [ 1322.210570] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1322.367287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1322.399563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.103@tcp [ 1322.427099] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1322.454806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1322.494315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.103@tcp -r 1 -m GET -l 3 [ 1322.530483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.103@tcp -r 1 -m GET -e local_error [ 1322.569904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.103@tcp -r 1 -m PUT -l 6 [ 1323.106332] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1323.623408] Lustre: DEBUG MARKER: Force oleg203-client.virtnet to discover 192.168.202.103@tcp (in background) [ 1323.706277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.103@tcp [ 1324.737987] Lustre: DEBUG MARKER: Force oleg203-server to discover 192.168.202.3@tcp [ 1326.725554] Lustre: DEBUG MARKER: Wait for 70478 [ 1326.757193] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1327.878362] Lustre: DEBUG MARKER: Finished wait on 70478 [ 1328.007845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1328.028779] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1328.035357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1328.260349] LNet: 70933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1328.264868] LNetError: 70933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1328.291157] LNet: Removed LNI 192.168.202.3@tcp [ 1328.624192] Key type .llcrypt unregistered [ 1328.626099] Key type ._llcrypt unregistered [ 1339.625671] Key type ._llcrypt registered [ 1339.627697] Key type .llcrypt registered [ 1340.655523] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 13:08:46 (1763402926) [ 1341.152297] Key type .llcrypt unregistered [ 1341.154077] Key type ._llcrypt unregistered [ 1345.200480] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1345.362215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1345.364794] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1345.370105] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1347.763188] Key type ._llcrypt registered [ 1347.764536] Key type .llcrypt registered [ 1347.929951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1347.943738] alg: No test for adler32 (adler32-zlib) [ 1348.872106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1348.914259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1348.940970] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1349.660593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1349.679846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1356.077344] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 13:09:01 (1763402941) [ 1356.220230] LNet: 73004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.227083] LNetError: 73004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.232295] LNet: Removed LNI 192.168.202.3@tcp [ 1356.241419] LNet: Removed LNI 10.1.2.3@tcp [ 1356.695918] Key type .llcrypt unregistered [ 1356.697510] Key type ._llcrypt unregistered [ 1360.903878] Key type ._llcrypt registered [ 1360.907410] Key type .llcrypt registered [ 1361.113850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1361.129322] alg: No test for adler32 (adler32-zlib) [ 1362.001435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.059682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.080610] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1362.699754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1362.719490] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1363.528327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.3@tcp [ 1364.575229] LNetError: Unexpected error -22 connecting to 192.168.202.3@tcp at host 192.168.202.3:988 [ 1368.922468] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 13:09:14 (1763402954) [ 1369.113625] LNet: 74132:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1369.120753] LNetError: 74132:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1369.126934] LNet: Removed LNI 192.168.202.3@tcp [ 1369.137868] LNet: Removed LNI 10.1.2.3@tcp [ 1369.598692] Key type .llcrypt unregistered [ 1369.600583] Key type ._llcrypt unregistered [ 1375.017953] Key type ._llcrypt registered [ 1375.019369] Key type .llcrypt registered [ 1375.210541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1375.220107] alg: No test for adler32 (adler32-zlib) [ 1376.171464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.235973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1376.254382] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1376.885710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1376.916672] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1377.687814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.3@tcp1 --nid 192.168.202.3@tcp2 [ 1381.328883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp1 192.168.202.3@tcp2 [ 1381.348136] LNetError: 74964:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.3@tcp1 to 192.168.202.3@tcp2 - Route table may be misconfigured [ 1381.367595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp2 192.168.202.3@tcp1 [ 1386.744102] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 13:09:32 (1763402972) [ 1386.838940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1386.854304] LNet: 75533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1386.865093] LNetError: 75533:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.874259] LNet: Removed LNI 192.168.202.3@tcp1 [ 1387.879323] LNetError: 75533:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.894535] LNet: Removed LNI 192.168.202.3@tcp2 [ 1387.920270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.994683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1388.029711] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1388.747731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.783429] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1389.536901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1389.573538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.3@tcp [ 1389.600636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp1 192.168.202.3@tcp1 [ 1389.641417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.3@tcp -d 192.168.202.3@tcp -e network_timeout [ 1389.677259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -e network_timeout [ 1389.707417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -e network_timeout [ 1389.732066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -e network_timeout [ 1389.776412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1389.787262] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1389.804266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1389.870632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1389.935608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1389.984648] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1395.184115] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 13:09:40 (1763402980) [ 1395.370864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1395.389738] LNet: 76400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1395.394851] LNetError: 76400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1395.400509] LNet: Removed LNI 192.168.202.3@tcp [ 1396.455866] LNet: Removed LNI 192.168.202.3@tcp1 [ 1396.470100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.540119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1396.717410] LNet: 76436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.963078] Key type .llcrypt unregistered [ 1396.964668] Key type ._llcrypt unregistered [ 1405.014795] Key type ._llcrypt registered [ 1405.016184] Key type .llcrypt registered [ 1406.190257] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 13:09:51 (1763402991) [ 1406.726068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.747462] alg: No test for adler32 (adler32-zlib) [ 1407.645528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1408.699706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1410.141888] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 13:09:55 (1763402995) [ 1410.327892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1410.366872] LNet: 77531:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.392306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1410.428537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1410.461872] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1411.127115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1411.143849] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1411.868384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1411.904960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp1 [ 1411.939074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp1 [ 1417.051795] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 13:10:02 (1763403002) [ 1418.598057] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg203-server [ 1420.312279] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 13:10:05 (1763403005) [ 1421.689824] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg203-server [ 1423.113801] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 13:10:08 (1763403008) [ 1424.419200] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg203-server [ 1425.625424] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 13:10:11 (1763403011) [ 1427.000683] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg203-server [ 1428.408427] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 13:10:14 (1763403014) [ 1429.968879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg203-server [ 1431.672536] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 13:10:17 (1763403017) [ 1433.387765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg203-server [ 1435.452459] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 13:10:20 (1763403020) [ 1437.306176] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg203-server [ 1438.841756] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 13:10:24 (1763403024) [ 1440.495065] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg203-server [ 1442.146855] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 13:10:27 (1763403027) [ 1443.687464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg203-server [ 1445.214133] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 13:10:30 (1763403030) [ 1445.369267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.437320] LNet: 81405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.442895] LNetError: 81405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.504031] LNet: Removed LNI 192.168.202.3@tcp [ 1446.519751] LNetError: 81405:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.536263] LNet: Removed LNI 192.168.202.3@tcp1 [ 1446.568714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.648691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.671456] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1447.341325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1447.420692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1447.651129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1447.667416] LNet: 81555:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.671553] LNetError: 81555:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.747796] LNet: Removed LNI 192.168.202.3@tcp [ 1448.775568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.827962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1448.863710] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1449.675453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1450.985948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.006786] LNet: 81709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.016666] LNetError: 81709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1452.069481] LNet: Removed LNI 192.168.202.3@tcp [ 1452.117506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1452.190937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1452.220118] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1452.840585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1452.900364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1453.149367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.181627] LNet: Removed LNI 192.168.202.3@tcp [ 1454.208321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.263357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1454.295606] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1454.930923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1455.121937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.141485] LNet: 82002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.146313] LNet: 82002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1455.155590] LNetError: 82002:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.160752] LNetError: 82002:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1455.174995] LNet: Removed LNI 192.168.202.3@tcp [ 1455.188558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.232931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.252210] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1455.800614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1455.862911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1456.052244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1457.138125] LNet: Removed LNI 192.168.202.3@tcp [ 1457.152414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.207848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1457.250989] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1457.835171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1458.023419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.110722] LNet: Removed LNI 192.168.202.3@tcp [ 1459.137079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.182229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1459.201289] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1459.757552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1459.824296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1461.032140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1462.117346] LNet: Removed LNI 192.168.202.3@tcp [ 1462.136968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.198827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1462.219366] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1462.754794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1462.932777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.978292] LNet: Removed LNI 192.168.202.3@tcp [ 1463.996157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.039478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1464.066549] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1464.528702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1464.583363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1464.744867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.768388] LNet: 82743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.773465] LNet: 82743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1464.781598] LNetError: 82743:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.786885] LNetError: 82743:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1464.797806] LNet: Removed LNI 192.168.202.3@tcp [ 1464.813503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.853928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1464.871927] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1465.428622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1466.821672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1467.880428] LNet: Removed LNI 192.168.202.3@tcp [ 1467.901836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.959535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1467.984429] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1468.656947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1468.728083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1470.146956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.205559] LNet: Removed LNI 192.168.202.3@tcp [ 1471.234466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1471.329784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1471.370156] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1472.138994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1472.385995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1473.450575] LNet: Removed LNI 192.168.202.3@tcp [ 1473.484400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.548972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1473.586252] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1474.215388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1474.262965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.289510] LNet: Removed LNI 192.168.202.3@tcp [ 1474.317744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.365904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1474.394851] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1474.974528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1480.286778] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 13:11:05 (1763403065) [ 1480.411877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1480.449913] LNet: Removed LNI 192.168.202.3@tcp [ 1480.474467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1480.517322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1480.540329] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 1481.450806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1481.464816] LNet: 84175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.468962] LNet: 84175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1481.475698] LNetError: 84175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.479134] LNetError: 84175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1481.493764] LNet: Removed LNI 192.168.202.3@tcp231 [ 1481.511633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.560826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1481.578364] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 1481.584312] LNet: 84183:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1481.900636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1481.950238] LNet: Removed LNI 192.168.202.3@tcp231 [ 1481.960942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1481.975751] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 1482.933714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.959660] LNet: Removed LNI 192.168.202.3@tcp231 [ 1482.976792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.033414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1483.057695] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/0] [ 1483.062211] LNet: 84335:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1483.520571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1483.545258] LNet: Removed LNI 192.168.202.3@tcp231 [ 1483.559089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1483.581823] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/0] [ 1484.458827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1484.485763] LNet: Removed LNI 192.168.202.3@tcp231 [ 1484.500314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.560110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1484.585787] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/60] [ 1484.591548] LNet: 84486:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1484.892188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1484.914561] LNet: Removed LNI 192.168.202.3@tcp231 [ 1484.929764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1484.948640] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/60] [ 1485.904551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1485.934521] LNet: Removed LNI 192.168.202.3@tcp231 [ 1485.953162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1486.026367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1486.054657] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 1486.360318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1486.385556] LNet: Removed LNI 192.168.202.3@tcp231 [ 1486.398951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1486.422497] LNet: Added LNI 192.168.202.3@tcp231 [8/256/0/180] [ 1491.867114] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 13:11:17 (1763403077) [ 1492.082152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.107631] LNet: Removed LNI 192.168.202.3@tcp231 [ 1492.124525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.169391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1492.189770] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 1492.722290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1492.754403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1492.779565] LNet: Removed LNI 192.168.202.3@tcp232 [ 1492.793630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1492.809825] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 1493.313785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1493.343787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1493.376621] LNet: Removed LNI 192.168.202.3@tcp232 [ 1493.390312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1493.420909] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 1493.968580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1493.990940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1494.017973] LNet: Removed LNI 192.168.202.3@tcp232 [ 1494.038339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1494.060017] LNet: Added LNI 192.168.202.3@tcp232 [8/256/0/180] [ 1494.577967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1494.604504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1494.632589] LNet: Removed LNI 192.168.202.3@tcp232 [ 1494.643721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1499.337789] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 13:11:25 (1763403085) [ 1500.133324] Key type .llcrypt unregistered [ 1500.139701] Key type ._llcrypt unregistered [ 1503.829670] Key type ._llcrypt registered [ 1503.835124] Key type .llcrypt registered [ 1504.034881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1504.052414] alg: No test for adler32 (adler32-zlib) [ 1504.953562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1504.992358] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1510.468213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1515.930270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.978141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1515.994459] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1517.200605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_interrupt -r 2 [ 1517.230276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 2 [ 1517.270122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_interrupt -r 2 [ 1517.307930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 2 [ 1517.339648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_interrupt -r 2 [ 1517.383536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_interrupt -r 2 [ 1517.429255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_interrupt -r 2 [ 1517.455780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_interrupt -r 2 [ 1517.489640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 1517.506377] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1517.770772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1517.796130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1517.829758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1517.860178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1517.901775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1517.953308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1518.008324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1518.043474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1518.062324] LNet: 87132:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1518.076581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1518.107860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1518.145568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1518.179549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1518.216498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1518.249815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1518.787993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_dropped -r 2 [ 1518.816476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 2 [ 1518.857610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_dropped -r 2 [ 1518.890847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 2 [ 1518.923089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_dropped -r 2 [ 1518.961428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_dropped -r 2 [ 1519.004490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_dropped -r 2 [ 1519.037593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_dropped -r 2 [ 1519.087797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 1519.110683] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1519.120534] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1519.350729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1519.377576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1519.456798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1519.503963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1519.542774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1519.577469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1519.621889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1519.660874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1519.683428] LNet: 87252:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1519.700840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1519.731881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1519.772356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1519.810569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1519.847972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1519.884227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1520.347219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_aborted -r 2 [ 1520.384837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 2 [ 1520.407391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_aborted -r 2 [ 1520.440143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 2 [ 1520.463626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_aborted -r 2 [ 1520.490763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_aborted -r 2 [ 1520.519845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_aborted -r 2 [ 1520.549972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_aborted -r 2 [ 1520.590711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 1520.622660] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1520.861149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1520.889629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1520.920675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1520.966248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1521.006608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1521.050463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1521.093769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1521.146028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1521.164218] LNet: 87372:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1521.185380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1521.210431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1521.247621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1521.285451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1521.318434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1521.353727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1521.831287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_no_route -r 2 [ 1521.855425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 2 [ 1521.891744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_no_route -r 2 [ 1521.929243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 2 [ 1521.960459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_no_route -r 2 [ 1522.001929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_no_route -r 2 [ 1522.048499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_no_route -r 2 [ 1522.078571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_no_route -r 2 [ 1522.109115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 1522.348849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1522.371535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1522.398751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1522.419805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1522.457106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1522.498745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1522.530316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1522.557246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1522.583139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1522.613521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1522.646450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1522.668730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1522.697656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1522.732502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1523.124184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e local_timeout -r 2 [ 1523.154407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 2 [ 1523.182109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e local_timeout -r 2 [ 1523.222995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 2 [ 1523.250982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e local_timeout -r 2 [ 1523.275563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e local_timeout -r 2 [ 1523.303979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e local_timeout -r 2 [ 1523.334525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e local_timeout -r 2 [ 1523.358176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 1523.382938] LNet: There was an unexpected network error while writing to 192.168.202.103: rc = -22 [ 1523.386954] LNet: Skipped 1 previous similar message [ 1523.588971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1523.614255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1523.646057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1523.677618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1523.711957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1523.749201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1523.780949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1523.808173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1523.823914] LNet: 87612:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1523.827951] LNet: 87612:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1523.843791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1523.868273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1523.889825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1523.930206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1523.951625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1523.968624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1524.354866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp -m GET -e remote_dropped -r 2 [ 1524.386107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 2 [ 1524.413445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp -m GET -e remote_dropped -r 2 [ 1524.445371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 2 [ 1524.483639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp -m GET -e remote_dropped -r 2 [ 1524.520457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.103@tcp1 -m GET -e remote_dropped -r 2 [ 1524.572432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp -m GET -e remote_dropped -r 2 [ 1524.596965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.3@tcp1 -d 192.168.202.3@tcp1 -m GET -e remote_dropped -r 2 [ 1524.639771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.3@tcp 192.168.202.103@tcp [ 1524.663855] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 1524.863730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1524.892362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1524.911164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1524.939883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1524.962744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1524.992584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1525.025739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1525.058763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1525.112879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1525.143761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1525.162222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1525.181554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1525.214514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1525.253804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1528.854168] LNet: 87805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.861339] LNetError: 87805:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.865831] LNet: Removed LNI 192.168.202.3@tcp [ 1528.874952] LNet: Removed LNI 192.168.202.3@tcp1 [ 1529.160212] Key type .llcrypt unregistered [ 1529.162116] Key type ._llcrypt unregistered [ 1539.994516] Key type ._llcrypt registered [ 1539.996277] Key type .llcrypt registered [ 1541.047477] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 13:12:06 (1763403126) [ 1541.399601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.411574] alg: No test for adler32 (adler32-zlib) [ 1542.314509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.362662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1542.386490] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1543.261129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1543.290507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.3@tcp [ 1543.312889] LNetError: 88780:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 1543.378896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.3@tcp [ 1543.394382] LNet: 1 local NIs in recovery (showing 1): 192.168.202.3@tcp [ 1543.464372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1548.888116] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 13:12:14 (1763403134) [ 1548.993983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.046832] LNet: 89363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.052469] LNetError: 89363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.123373] LNet: Removed LNI 192.168.202.3@tcp [ 1550.148588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.210727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.243587] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1550.881817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1550.903034] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1551.686871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.3@tcp [ 1551.703846] LNetError: 89635:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.3@tcp to recovery [ 1552.229350] LNet: There was an unexpected network error while writing to 192.168.202.3: rc = -22 [ 1552.864763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.3@tcp [ 1560.037438] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 13:12:25 (1763403145) [ 1561.486880] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg203-server [ 1563.142569] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 13:12:28 (1763403148) [ 1563.309480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.339586] LNet: 90590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.344566] LNetError: 90590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1563.348440] LNet: Removed LNI 192.168.202.3@tcp [ 1563.360523] LNet: Removed LNI 192.168.202.3@tcp1 [ 1563.377275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.451061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.485079] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1564.053055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1564.096324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1564.130987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1564.197540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1564.252494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1564.310974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1568.750371] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 13:12:34 (1763403154) [ 1568.933582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.961561] LNet: 91315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.970949] LNetError: 91315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.979083] LNetError: 91315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1569.000842] LNet: Removed LNI 192.168.202.3@tcp [ 1569.020825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.099133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.128973] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1574.919699] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 13:12:40 (1763403160) [ 1575.024792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.049791] LNet: 92017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.053696] LNetError: 92017:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.072964] LNet: Removed LNI 192.168.202.3@tcp [ 1575.086402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.127527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.150709] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1581.169139] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 13:12:46 (1763403166) [ 1582.397502] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1583.996836] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 13:12:49 (1763403169) [ 1584.370488] LNet: 93052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.374955] LNetError: 93052:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.394709] LNet: Removed LNI 192.168.202.3@tcp [ 1584.893100] Key type .llcrypt unregistered [ 1584.894833] Key type ._llcrypt unregistered [ 1589.740695] Key type ._llcrypt registered [ 1589.745748] Key type .llcrypt registered [ 1589.959867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1589.980448] alg: No test for adler32 (adler32-zlib) [ 1590.917747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1590.948370] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1597.025927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1601.571042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.103@tcp [ 1601.864782] LNet: 93464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1601.870666] LNetError: 93464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1601.882789] LNet: Removed LNI 192.168.202.3@tcp [ 1602.205738] Key type .llcrypt unregistered [ 1602.208188] Key type ._llcrypt unregistered [ 1610.756924] Key type ._llcrypt registered [ 1610.758758] Key type .llcrypt registered [ 1611.817245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1613.053131] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1614.551193] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 13:13:20 (1763403200) [ 1614.958601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1614.973709] alg: No test for adler32 (adler32-zlib) [ 1615.897213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.978227] LNet: 94547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.502384] Key type .llcrypt unregistered [ 1616.504056] Key type ._llcrypt unregistered [ 1620.611659] Key type ._llcrypt registered [ 1620.613300] Key type .llcrypt registered [ 1620.794622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1620.818468] alg: No test for adler32 (adler32-zlib) [ 1621.740877] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1621.743225] LNet: 94799:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1621.776069] LNetError: 94811:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 1621.785897] LNetError: 94811:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.6@tcp has gone from up to down [ 1622.160599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.185194] LNet: 94857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.192978] LNetError: 94857:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.207842] LNet: Removed LNI 192.168.202.3@tcp [ 1622.226821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.281801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1622.312197] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1622.320349] LNet: 94864:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1622.325755] LNet: 94864:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1622.338580] LNetError: 94864:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.4@tcp is a gateway. Can not delete it [ 1622.343836] LNetError: 94864:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.202.4@tcp is a router. Can not be deleted [ 1622.368111] LNetError: 94861:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.6@tcp is being used as a gateway but routing feature is not turned on [ 1622.381235] LNetError: 94861:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 1622.392088] LNetError: 94861:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.6@tcp has gone from up to down [ 1622.407960] LNetError: 94861:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1622.422431] LNet: 94864:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.795842] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 13:13:33 (1763403213) [ 1629.555645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg203-server [ 1630.901588] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 13:13:36 (1763403216) [ 1632.628324] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg203-server [ 1634.389982] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 13:13:39 (1763403219) [ 1635.729596] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 1637.703200] Key type lgssc registered [ 1638.746266] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1654.703117] Key type lgssc unregistered [ 1654.956092] LNet: 97282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.969199] LNetError: 97282:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.997485] LNet: Removed LNI 192.168.202.3@tcp [ 1655.403247] Key type .llcrypt unregistered [ 1655.404954] Key type ._llcrypt unregistered [ 1667.180962] Key type ._llcrypt registered [ 1667.187376] Key type .llcrypt registered [ 1668.187651] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 13:14:13 (1763403253) [ 1668.465549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1668.477243] alg: No test for adler32 (adler32-zlib) [ 1669.359272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.397995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1669.427833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1669.480228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1669.525569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1674.755811] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 13:14:20 (1763403260) [ 1674.873235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.929362] LNet: 98678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.980633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.063952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1675.104090] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1675.910589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.950358] LNet: 98829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.955923] LNetError: 98829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1675.972578] LNet: Removed LNI 192.168.202.3@tcp [ 1675.984542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.075712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1676.120927] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1676.802824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.828539] LNetError: 98857:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.840878] LNet: Removed LNI 192.168.202.3@tcp [ 1676.851033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.933875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1676.971073] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1683.824158] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 13:14:28 (1763403268) [ 1683.934354] LNet: 99441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.937161] LNet: 99441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1683.962235] LNetError: 99441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.990989] LNet: Removed LNI 192.168.202.3@tcp [ 1684.949295] Key type .llcrypt unregistered [ 1684.953389] Key type ._llcrypt unregistered [ 1691.135160] Key type ._llcrypt registered [ 1691.136682] Key type .llcrypt registered [ 1691.388332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1691.401300] alg: No test for adler32 (adler32-zlib) [ 1692.307885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.412095] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1692.414594] LNet: *** cfs_fail_loc=e003, val=100*** [ 1693.143743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1693.170846] LNet: *** cfs_fail_loc=e003, val=100*** [ 1693.174408] LNet: Skipped 2 previous similar messages [ 1693.197278] Ping reply received with mapping: [ 1693.199097] found metadata reply NI[192.168.202.3@tcp] [ 1693.204031] 03 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1693.206277] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1693.208440] 5a 5a 5a 5a 5a 5a 5a 5a [ 1693.393935] LNet: 99842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.397849] LNetError: 99842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.417783] LNet: Removed LNI 192.168.202.3@tcp [ 1694.020478] Key type .llcrypt unregistered [ 1694.024669] Key type ._llcrypt unregistered [ 1703.455860] Key type ._llcrypt registered [ 1703.457525] Key type .llcrypt registered [ 1704.611103] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 13:14:50 (1763403290) [ 1705.152368] Key type .llcrypt unregistered [ 1705.154740] Key type ._llcrypt unregistered [ 1710.265706] Key type ._llcrypt registered [ 1710.267469] Key type .llcrypt registered [ 1710.420520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1710.436654] alg: No test for adler32 (adler32-zlib) [ 1711.448642] LNetError: 100877:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1711.735389] Key type .llcrypt unregistered [ 1711.738246] Key type ._llcrypt unregistered [ 1716.213872] Key type ._llcrypt registered [ 1716.215523] Key type .llcrypt registered [ 1717.750360] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 13:15:02 (1763403302) [ 1718.335645] Key type .llcrypt unregistered [ 1718.337525] Key type ._llcrypt unregistered [ 1723.700844] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1723.733348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1726.375975] Key type ._llcrypt registered [ 1726.377550] Key type .llcrypt registered [ 1726.589776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1726.599623] alg: No test for adler32 (adler32-zlib) [ 1727.577458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.671922] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1738.808653] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 13:15:23 (1763403323) [ 1738.902590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1738.934611] LNet: 102642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.942104] LNetError: 102642:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1738.958125] LNet: Removed LNI 10.1.2.3@tcp [ 1738.984748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.698220] Key type .llcrypt unregistered [ 1739.699751] Key type ._llcrypt unregistered [ 1744.515344] Key type ._llcrypt registered [ 1744.517986] Key type .llcrypt registered [ 1744.612718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1744.623554] alg: No test for adler32 (adler32-zlib) [ 1745.556619] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1745.614749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1745.736847] LNet: 102934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1745.747443] LNetError: 102934:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1745.765969] LNet: Removed LNI 192.168.202.3@tcp [ 1746.266492] Key type .llcrypt unregistered [ 1746.267955] Key type ._llcrypt unregistered [ 1755.146816] Key type ._llcrypt registered [ 1755.148399] Key type .llcrypt registered [ 1756.354881] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 13:15:41 (1763403341) [ 1756.842955] Key type .llcrypt unregistered [ 1756.845563] Key type ._llcrypt unregistered [ 1761.541546] Key type ._llcrypt registered [ 1761.543311] Key type .llcrypt registered [ 1761.735751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1761.747566] alg: No test for adler32 (adler32-zlib) [ 1767.837141] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 13:15:53 (1763403353) [ 1767.960353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.012815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.113373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.138588] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1768.859386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.886211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1768.909583] LNet: Removed LNI 192.168.202.3@tcp [ 1769.130936] LNet: 104669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1769.466630] Key type .llcrypt unregistered [ 1769.468161] Key type ._llcrypt unregistered [ 1778.256359] Key type ._llcrypt registered [ 1778.261510] Key type .llcrypt registered [ 1779.653282] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 13:16:04 (1763403364) [ 1780.056969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.073155] alg: No test for adler32 (adler32-zlib) [ 1780.973973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.021150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.046710] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1781.667532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1781.697285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.3@tcp [ 1781.713950] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1781.717135] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1781.719392] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1781, 0, 0 [ 1781.728443] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1781.733062] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1781.739911] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1781.745501] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1781, 0, 0 [ 1781.754704] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1781.758987] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1781.762785] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1781.765508] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1781, 0, 1831 [ 1781.769633] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1781.772082] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1781.775292] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1781.778329] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1781, 0, 1831 [ 1781.782404] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1787.372095] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 13:16:12 (1763403372) [ 1787.771160] LNet: 106205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.779428] LNetError: 106205:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.812919] LNet: Removed LNI 192.168.202.3@tcp [ 1788.411244] Key type .llcrypt unregistered [ 1788.412606] Key type ._llcrypt unregistered [ 1792.434340] Key type ._llcrypt registered [ 1792.436103] Key type .llcrypt registered [ 1792.690272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1792.720348] alg: No test for adler32 (adler32-zlib) [ 1793.655214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1793.682493] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1800.038940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1807.163860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1807.191635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1807.228487] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 1808.281945] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1808.478900] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1808.487078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1808.504302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1810.847468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1810.902023] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1822.231623] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 13:16:47 (1763403407) [ 1822.324754] LNet: 107699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.329154] LNetError: 107699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.333957] LNet: Removed LNI 192.168.202.3@tcp [ 1823.391508] LNet: Removed LNI 192.168.202.3@tcp1 [ 1823.393263] LNetError: 107699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1823.414731] LNetError: 107699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1823.431940] LNet: Removed LNI 10.1.2.3@tcp99 [ 1824.068268] Key type .llcrypt unregistered [ 1824.069860] Key type ._llcrypt unregistered [ 1829.571277] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1829.580186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1832.016083] Key type ._llcrypt registered [ 1832.017279] Key type .llcrypt registered [ 1832.160319] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1832.173092] alg: No test for adler32 (adler32-zlib) [ 1833.070874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.118551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.140070] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1833.696417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1833.709208] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1835.478988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1835.520438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1835.563351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1835.577938] LNetError: 108447:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1835.610835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1835.648285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1835.973400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1836.029394] LNet: 108467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1836.033258] LNetError: 108467:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1836.038833] LNet: Removed LNI 192.168.202.3@tcp [ 1836.052884] LNet: Removed LNI 10.1.2.3@tcp [ 1836.541643] Key type .llcrypt unregistered [ 1836.543446] Key type ._llcrypt unregistered [ 1845.666351] Key type ._llcrypt registered [ 1845.667840] Key type .llcrypt registered [ 1846.727111] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 13:17:12 (1763403432) [ 1847.151079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.160725] alg: No test for adler32 (adler32-zlib) [ 1848.064793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1848.106659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1848.131561] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1848.708390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.3@tcp [ 1848.794933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg203-client@tcp [ 1853.822102] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 13:17:19 (1763403439) [ 1853.931421] LNet: 109994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.941265] LNetError: 109994:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.961320] LNet: Removed LNI 192.168.202.3@tcp [ 1854.439691] Key type .llcrypt unregistered [ 1854.441484] Key type ._llcrypt unregistered [ 1858.931776] Key type ._llcrypt registered [ 1858.933198] Key type .llcrypt registered [ 1859.103680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1859.123351] alg: No test for adler32 (adler32-zlib) [ 1859.972023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1860.003328] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1860.026095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1865.263750] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 13:17:30 (1763403450) [ 1866.630722] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1868.254878] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 13:17:33 (1763403453) [ 1869.316591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1870.632467] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 13:17:36 (1763403456) [ 1870.711937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.728422] LNet: 111406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.732630] LNetError: 111406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1870.753569] LNet: Removed LNI 192.168.202.3@tcp [ 1870.779629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.830466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.857175] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1871.407522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1871.447516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1876.428144] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 13:17:42 (1763403462) [ 1876.483491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.500406] LNet: 112114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.504730] LNetError: 112114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.542191] LNet: Removed LNI 192.168.202.3@tcp [ 1877.564631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1877.614359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1877.650516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1877.806782] LNet: 112156:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.048492] Key type .llcrypt unregistered [ 1878.050271] Key type ._llcrypt unregistered [ 1884.654632] Key type ._llcrypt registered [ 1884.656594] Key type .llcrypt registered [ 1885.827438] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 13:17:51 (1763403471) [ 1886.114196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1886.135659] alg: No test for adler32 (adler32-zlib) [ 1887.039798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1887.074616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1887.090587] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1887.670596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1887.738467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1893.357198] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 13:17:58 (1763403478) [ 1893.474514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.507381] LNet: 113669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.513421] LNetError: 113669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.537337] LNet: Removed LNI 192.168.202.3@tcp [ 1893.564266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.608885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1893.634518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1899.469200] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 13:18:04 (1763403484) [ 1899.713929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1899.756332] LNet: 114252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.785790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.848556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1900.428167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1900.456883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1900.521637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1901.022632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1901.080894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1901.193922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.215082] LNet: 114285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.238934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.280668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1901.744025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1901.792845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1901.867436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1902.293555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1902.349044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1908.193590] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 13:18:13 (1763403493) [ 1908.426672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1908.466765] LNet: 114885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1908.515248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.562733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1908.600352] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1909.275161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1909.291940] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1910.048851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1910.116482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1910.316763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.333900] LNetError: 115165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.391128] LNet: Removed LNI 192.168.202.3@tcp [ 1911.396478] LNetError: 115165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.408860] LNet: Removed LNI 192.168.202.3@tcp2 [ 1911.436269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.494363] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1911.504925] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1911.511604] LNet: 115171:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1911.882839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1912.025748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1913.119154] LNet: Removed LNI 192.168.202.3@tcp [ 1913.136408] LNet: Removed LNI 192.168.202.3@tcp2 [ 1913.166161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.245335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.282537] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1914.038189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1914.057219] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1914.835746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1914.866821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1914.908342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1915.082648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.102909] LNet: 115492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.107668] LNet: 115492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1915.120389] LNetError: 115492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.126993] LNetError: 115492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1915.136020] LNet: Removed LNI 192.168.202.3@tcp [ 1916.204684] LNet: Removed LNI 192.168.202.3@tcp2 [ 1916.246830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.342065] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1916.346094] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1916.349601] LNet: 115499:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1916.892204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1917.183526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.219805] LNet: Removed LNI 192.168.202.3@tcp [ 1918.245927] LNet: Removed LNI 192.168.202.3@tcp2 [ 1918.297220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.383944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.421236] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1919.255037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1919.277300] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1920.246666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1920.308634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1920.532378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.561244] LNetError: 115815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.570372] LNetError: 115815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1920.578973] LNet: Removed LNI 192.168.202.3@tcp [ 1920.605857] LNet: Removed LNI 192.168.202.3@tcp2 [ 1920.627236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.690748] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1920.695819] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1920.709891] LNet: 115821:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1921.474922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1921.733961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1922.783277] LNet: Removed LNI 192.168.202.3@tcp [ 1922.802461] LNet: Removed LNI 192.168.202.3@tcp2 [ 1922.839630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.878172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.905445] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1923.548457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1923.564951] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1924.632301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1924.670806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1924.743658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1924.997968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.035174] LNet: 116143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.039372] LNet: 116143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1926.111255] LNet: Removed LNI 192.168.202.3@tcp [ 1927.145739] LNet: Removed LNI 192.168.202.3@tcp2 [ 1927.175161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.266933] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1927.273567] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1927.279598] LNet: 116149:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1927.746817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1928.012044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1928.036052] LNet: Removed LNI 192.168.202.3@tcp [ 1929.065552] LNet: Removed LNI 192.168.202.3@tcp2 [ 1929.105733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.173195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.221385] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1930.033232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1930.049785] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1930.765252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1930.789677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1930.832595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1931.144814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.179241] LNetError: 116483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.183229] LNetError: 116483:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1932.191192] LNet: Removed LNI 192.168.202.3@tcp [ 1932.207415] LNet: Removed LNI 192.168.202.3@tcp2 [ 1932.236900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.320754] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1932.326667] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1932.331456] LNet: 116489:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1932.685905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1933.089251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1934.175253] LNet: Removed LNI 192.168.202.3@tcp [ 1934.196398] LNet: Removed LNI 192.168.202.3@tcp2 [ 1934.245509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1934.330429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1934.382887] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1935.274261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1935.289480] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1936.174460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1936.219392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1936.254036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1936.303943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1936.627441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.695198] LNet: Removed LNI 192.168.202.3@tcp [ 1938.729554] LNet: Removed LNI 192.168.202.3@tcp2 [ 1938.747383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.795642] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1938.804736] LNet: Added LNI 192.168.202.3@tcp2 [8/256/0/180] [ 1939.123371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1944.558895] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1945.941397] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 13:18:51 (1763403531) [ 1946.268728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.297908] LNet: 117607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.302614] LNet: 117607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1946.313413] LNet: Removed LNI 192.168.202.3@tcp [ 1947.387279] LNet: Removed LNI 192.168.202.3@tcp2 [ 1947.411091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.669627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.717177] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1948.578171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1948.604735] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1949.498903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.520968] LNetError: 117880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.524593] LNetError: 117880:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1949.530580] LNet: Removed LNI 192.168.202.3@tcp [ 1949.538739] LNet: Removed LNI 10.1.2.3@tcp [ 1949.580733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.645264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1949.664659] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1949.667616] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1949.931252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1949.950932] LNet: Removed LNI 192.168.202.3@tcp [ 1949.964735] LNet: Removed LNI 10.1.2.3@tcp [ 1955.821442] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 13:19:01 (1763403541) [ 1955.894709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.936706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.059155] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1956.129489] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1956.136242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1956.144677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1957.448698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.477505] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1958.117794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1958.138505] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1959.780495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.3@tcp [ 1959.823994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1959.908630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.3@tcp -m PUT -e local_timeout -r 1 [ 1959.992852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1960.066460] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1960.076509] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1960.300798] LNetError: 118538:0:(lib-move.c:4947:lnet_parse()) 192.168.202.3@tcp, src 192.168.202.3@tcp: Dropping PUT (error -108 looking up sender) [ 1962.336137] LNet: Removed LNI 192.168.202.3@tcp [ 1962.350347] LNet: Removed LNI 10.1.2.3@tcp [ 1962.726996] Key type .llcrypt unregistered [ 1962.728994] Key type ._llcrypt unregistered [ 1972.753855] Key type ._llcrypt registered [ 1972.755542] Key type .llcrypt registered [ 1974.045180] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 13:19:19 (1763403559) [ 1974.604513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1974.643989] alg: No test for adler32 (adler32-zlib) [ 1975.569335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.914909] LNet: 119846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.382332] Key type .llcrypt unregistered [ 1976.385174] Key type ._llcrypt unregistered [ 1981.186606] Key type ._llcrypt registered [ 1981.190845] Key type .llcrypt registered [ 1981.427222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1981.445565] alg: No test for adler32 (adler32-zlib) [ 1982.355333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1982.402956] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 1988.418548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 1988.611285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1989.263156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.103@tcp [ 1989.468551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1989.505704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1989.555490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1989.589223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1989.601462] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 1989.618130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1989.657904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1989.698533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1989.735984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1989.751539] LNet: 120291:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1989.777061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1989.807485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1989.849947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1989.870533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1989.906710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1989.938133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1990.418530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1990.617607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1990.648554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1990.672847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1990.706650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1990.743988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1990.770394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1990.791585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1990.821304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1990.832154] LNet: 120408:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1990.843863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1990.878562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1990.904415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1990.925912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1990.949520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1990.980148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1991.475669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.103@tcp [ 1991.665967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1991.690586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1991.723091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1991.738778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1991.777698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1991.812953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1991.838677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1991.865457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1991.878447] LNet: 120526:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1991.891161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1991.921576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1991.946748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1991.973916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1991.999247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1992.031867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1992.472686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1992.637526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1992.665972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1992.687807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1992.716702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1992.747956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1992.771336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1992.796148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1992.819806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1992.852905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1992.890702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1992.921471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1992.954896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1992.986837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1993.014216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1993.299956] LNet: 120688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1993.308269] LNetError: 120688:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1993.317514] LNet: Removed LNI 192.168.202.3@tcp [ 1993.654772] Key type .llcrypt unregistered [ 1993.656406] Key type ._llcrypt unregistered [ 2006.468731] Key type ._llcrypt registered [ 2006.470158] Key type .llcrypt registered [ 2007.482601] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 13:19:53 (1763403593) [ 2007.837983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2007.853925] alg: No test for adler32 (adler32-zlib) [ 2008.785210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.283928] LNet: 121572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.846540] Key type .llcrypt unregistered [ 2009.849514] Key type ._llcrypt unregistered [ 2014.733393] Key type ._llcrypt registered [ 2014.740408] Key type .llcrypt registered [ 2015.098564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.118391] alg: No test for adler32 (adler32-zlib) [ 2016.088977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2016.112606] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 2023.483358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.103@tcp [ 2031.590785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.656748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2031.682158] LNet: Added LNI 192.168.202.3@tcp1 [8/256/0/180] [ 2032.662764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2033.257811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.103@tcp [ 2033.486912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2033.516344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2033.547485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2033.563415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2033.579656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2033.595438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2033.641937] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.103@tcp [ 2033.662617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2033.697617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2033.714249] LNet: 122220:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2033.732769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2033.754649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2033.782105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2033.799915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2033.834604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2033.866452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2034.529585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2034.797118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2034.849737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2034.889156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2034.916522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2034.945510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2034.973852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2034.994271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2035.020622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2035.035933] LNet: 122337:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2035.062382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2035.100436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2035.170613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2035.226264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2035.243338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2035.272882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2035.725406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.103@tcp [ 2035.905462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2035.927195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2035.948472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2035.973161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2035.996947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2036.032256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2036.060536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2036.097232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2036.108386] LNet: 122454:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2036.125636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2036.150125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2036.176770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2036.210466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2036.249366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2036.277656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2036.764232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2036.982271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2037.012781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2037.065613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2037.135183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2037.194495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2037.232532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2037.257728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2037.280736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2037.299758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2037.355374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2037.383767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2037.401929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2037.432719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2037.476693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2041.420883] LNet: 122651:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.428974] LNetError: 122651:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.442078] LNet: Removed LNI 192.168.202.3@tcp [ 2041.458798] LNet: Removed LNI 192.168.202.3@tcp1 [ 2041.912840] Key type .llcrypt unregistered [ 2041.919996] Key type ._llcrypt unregistered [ 2053.177052] Key type ._llcrypt registered [ 2053.179458] Key type .llcrypt registered [ 2054.046686] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1783 sec ========== 13:20:39 (1763403639) [ 2054.551459] Key type .llcrypt unregistered [ 2054.554504] Key type ._llcrypt unregistered [ 2062.735879] Key type ._llcrypt registered [ 2062.737547] Key type .llcrypt registered [ 2063.064787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.073351] alg: No test for adler32 (adler32-zlib) [ 2064.036941] Lustre: Lustre: Build Version: 2.16.61_42_g7dd025e [ 2064.201887] LNet: Added LNI 192.168.202.3@tcp [8/256/0/180] [ 2065.849168] Key type lgssc registered [ 2066.740228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2120.298036] Lustre: Mounted lustre-client [ 2125.122956] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2136.587646] LustreError: 125873:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8fbcd08aa000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2136.613500] LustreError: 125873:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2136.645794] Lustre: Unmounted lustre-client [ 2153.974103] Key type lgssc unregistered [ 2154.215090] LNet: 126353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2154.220040] LNetError: 126353:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2154.238615] LNet: Removed LNI 192.168.202.3@tcp [ 2154.810446] Key type .llcrypt unregistered [ 2154.812124] Key type ._llcrypt unregistered