[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477509428 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002372] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.005009] kvm-guest: setup PV IPIs [ 0.007951] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.012038] Yama: becoming mindful. [ 0.013041] SELinux: Initializing. [ 0.014069] *** VALIDATE selinux *** [ 0.022522] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027349] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028152] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029108] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030118] *** VALIDATE tmpfs *** [ 0.032417] *** VALIDATE proc *** [ 0.033240] *** VALIDATE cgroup *** [ 0.034007] *** VALIDATE cgroup2 *** [ 0.035228] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036151] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038033] Spectre V2 : User space: Vulnerable [ 0.039009] Speculative Store Bypass: Vulnerable [ 0.042551] debug: unmapping init [mem 0xffffffffbde59000-0xffffffffbde60fff] [ 0.044164] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045795] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047009] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050023] ... max period: 00007fffffffffff [ 0.051009] ... fixed-purpose events: 3 [ 0.052008] ... event mask: 000000070000000f [ 0.053283] rcu: Hierarchical SRCU implementation. [ 0.055461] smp: Bringing up secondary CPUs ... [ 0.056607] x86: Booting SMP configuration: [ 0.057027] .... node #0, CPUs: #1 #2 #3 [ 0.061114] smp: Brought up 1 node, 4 CPUs [ 0.063014] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151028] node 0 deferred pages initialised in 86ms [ 0.156196] devtmpfs: initialized [ 0.157266] x86/mm: Memory block size: 128MB [ 0.159922] gcov: version magic: 0x41383552 [ 0.161474] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162109] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163338] pinctrl core: initialized pinctrl subsystem [ 0.164292] [ 0.164786] ************************************************************* [ 0.165024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166016] ** ** [ 0.167016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.168017] ** ** [ 0.169023] ** This means that this kernel is built to expose internal ** [ 0.170021] ** IOMMU data structures, which may compromise security on ** [ 0.171023] ** your system. ** [ 0.172028] ** ** [ 0.173025] ** If you see this message and you are not debugging the ** [ 0.174040] ** kernel, report this immediately to your vendor! ** [ 0.175023] ** ** [ 0.176021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.177014] ************************************************************* [ 0.178868] NET: Registered protocol family 16 [ 0.179586] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.180102] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181108] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.183022] cpuidle: using governor menu [ 0.184928] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.188540] PCI: Using configuration type 1 for base access [ 0.190134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.206043] cryptd: max_cpu_qlen set to 1000 [ 0.208284] ACPI: Added _OSI(Module Device) [ 0.210021] ACPI: Added _OSI(Processor Device) [ 0.212018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.213064] ACPI: Added _OSI(Processor Aggregator Device) [ 0.217869] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.222626] ACPI: Interpreter enabled [ 0.224056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.225010] ACPI: Using IOAPIC for interrupt routing [ 0.227101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.229354] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238317] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.242017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.248177] acpiphp: Slot [2] registered [ 0.249090] acpiphp: Slot [3] registered [ 0.250077] acpiphp: Slot [4] registered [ 0.251088] acpiphp: Slot [5] registered [ 0.253020] acpiphp: Slot [6] registered [ 0.254118] acpiphp: Slot [7] registered [ 0.255133] acpiphp: Slot [8] registered [ 0.257090] acpiphp: Slot [9] registered [ 0.258087] acpiphp: Slot [10] registered [ 0.259085] acpiphp: Slot [11] registered [ 0.261079] acpiphp: Slot [12] registered [ 0.262089] acpiphp: Slot [13] registered [ 0.263097] acpiphp: Slot [14] registered [ 0.264088] acpiphp: Slot [15] registered [ 0.266101] acpiphp: Slot [16] registered [ 0.267098] acpiphp: Slot [17] registered [ 0.269113] acpiphp: Slot [18] registered [ 0.270107] acpiphp: Slot [19] registered [ 0.272123] acpiphp: Slot [20] registered [ 0.273091] acpiphp: Slot [21] registered [ 0.275094] acpiphp: Slot [22] registered [ 0.276091] acpiphp: Slot [23] registered [ 0.277095] acpiphp: Slot [24] registered [ 0.279092] acpiphp: Slot [25] registered [ 0.280082] acpiphp: Slot [26] registered [ 0.281055] acpiphp: Slot [27] registered [ 0.282072] acpiphp: Slot [28] registered [ 0.284087] acpiphp: Slot [29] registered [ 0.285086] acpiphp: Slot [30] registered [ 0.286088] acpiphp: Slot [31] registered [ 0.288073] PCI host bridge to bus 0000:00 [ 0.289016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.291022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.295022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.297023] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.300026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305243] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309342] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.315936] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.319364] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.326021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.328361] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.331784] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.335043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336572] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.340957] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.350931] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.356016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.361564] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.374025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.388026] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.398424] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.408024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.413018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.425024] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.435636] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.438402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.441467] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.444544] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.447334] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.454158] iommu: Default domain type: Passthrough [ 0.456570] SCSI subsystem initialized [ 0.458177] ACPI: bus type USB registered [ 0.460166] usbcore: registered new interface driver usbfs [ 0.463142] usbcore: registered new interface driver hub [ 0.466138] usbcore: registered new device driver usb [ 0.468252] pps_core: LinuxPPS API ver. 1 registered [ 0.470016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.474088] PTP clock support registered [ 0.476057] EDAC MC: Ver: 3.0.0 [ 0.478143] PCI: Using ACPI for IRQ routing [ 0.483823] NetLabel: Initializing [ 0.485019] NetLabel: domain hash size = 128 [ 0.487017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.489140] NetLabel: unlabeled traffic allowed by default [ 0.491242] vgaarb: loaded [ 0.495502] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.496025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.503429] clocksource: Switched to clocksource kvm-clock [ 0.612153] VFS: Disk quotas dquot_6.6.0 [ 0.613982] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.616739] *** VALIDATE ramfs *** [ 0.617878] *** VALIDATE hugetlbfs *** [ 0.619647] pnp: PnP ACPI init [ 0.622371] pnp: PnP ACPI: found 6 devices [ 0.643667] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.646240] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.647985] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.649689] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.651844] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.655643] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.658499] NET: Registered protocol family 2 [ 0.660870] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.665077] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.668238] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.673715] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.677525] TCP: Hash tables configured (established 65536 bind 65536) [ 0.680283] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.683919] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.686925] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.689995] NET: Registered protocol family 1 [ 0.692708] RPC: Registered named UNIX socket transport module. [ 0.694817] RPC: Registered udp transport module. [ 0.696473] RPC: Registered tcp transport module. [ 0.698175] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.700715] NET: Registered protocol family 44 [ 0.702271] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.704398] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.706513] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.709039] PCI: CLS 0 bytes, default 64 [ 0.710803] Unpacking initramfs... [ 2.161819] debug: unmapping init [mem 0xffff90f27cc64000-0xffff90f27ffcffff] [ 2.165852] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.168195] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.171468] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.687362] Initialise system trusted keyrings [ 2.689249] Key type blacklist registered [ 2.693948] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.706211] zbud: loaded [ 2.709332] *** VALIDATE nfs *** [ 2.710417] *** VALIDATE nfs4 *** [ 2.713077] pstore: using deflate compression [ 2.719211] Platform Keyring initialized [ 2.833492] NET: Registered protocol family 38 [ 2.835700] Key type asymmetric registered [ 2.837148] Asymmetric key parser 'x509' registered [ 2.839193] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.842470] io scheduler mq-deadline registered [ 2.844223] io scheduler kyber registered [ 2.845874] io scheduler bfq registered [ 2.848426] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.851501] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.854208] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.856511] ACPI: Power Button [PWRF] [ 2.950323] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.040358] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.140688] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.175876] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.207542] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.216306] Non-volatile memory driver v1.3 [ 3.217890] Linux agpgart interface v0.103 [ 3.251683] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.254617] vda: detected capacity change from 0 to 68530176 [ 3.268146] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.270756] vdb: detected capacity change from 0 to 1073741824 [ 3.276178] libphy: Fixed MDIO Bus: probed [ 3.284533] usbcore: registered new interface driver usbserial_generic [ 3.287102] usbserial: USB Serial support registered for generic [ 3.289311] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.293494] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.295397] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.297552] mousedev: PS/2 mouse device common for all mice [ 3.300848] rtc_cmos 00:05: RTC can wake from S4 [ 3.304143] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.306034] rtc_cmos 00:05: registered as rtc0 [ 3.313135] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.315501] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.316579] intel_pstate: CPU model not supported [ 3.321416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.326087] hid: raw HID events driver (C) Jiri Kosina [ 3.328279] usbcore: registered new interface driver usbhid [ 3.330157] usbhid: USB HID core driver [ 3.331472] drop_monitor: Initializing network drop monitor service [ 3.333969] Initializing XFRM netlink socket [ 3.336593] NET: Registered protocol family 10 [ 3.340324] Segment Routing with IPv6 [ 3.341427] NET: Registered protocol family 17 [ 3.342980] mpls_gso: MPLS GSO support [ 3.349711] RAS: Correctable Errors collector initialized. [ 3.351960] AVX version of gcm_enc/dec engaged. [ 3.353776] AES CTR mode by8 optimization enabled [ 3.439354] sched_clock: Marking stable (3439330163, 0)->(4360298597, -920968434) [ 3.442400] registered taskstats version 1 [ 3.445554] Loading compiled-in X.509 certificates [ 3.447886] zswap: loaded using pool lzo/zbud [ 3.479252] Key type big_key registered [ 3.492737] Key type encrypted registered [ 3.494788] ima: No TPM chip found, activating TPM-bypass! [ 3.497102] ima: Allocated hash algorithm: sha1 [ 3.498671] ima: No architecture policies found [ 3.500463] evm: Initialising EVM extended attributes: [ 3.502504] evm: security.selinux [ 3.503722] evm: security.ima [ 3.504972] evm: security.capability [ 3.506595] evm: HMAC attrs: 0x1 [ 3.509870] rtc_cmos 00:05: setting system clock to 2025-11-17 01:15:49 UTC (1763342149) [ 3.518312] debug: unmapping init [mem 0xffffffffbee03000-0xffffffffbeffffff] [ 3.521871] debug: unmapping init [mem 0xffffffffbdb82000-0xffffffffbde58fff] [ 3.533154] Write protecting the kernel read-only data: 28672k [ 3.537103] debug: unmapping init [mem 0xffffffffbc203000-0xffffffffbc3fffff] [ 3.540290] debug: unmapping init [mem 0xffffffffbcb14000-0xffffffffbcbfffff] [ 3.574290] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.581984] systemd[1]: Detected virtualization kvm. [ 3.583809] systemd[1]: Detected architecture x86-64. [ 3.585419] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.610357] systemd[1]: No hostname configured. [ 3.612172] systemd[1]: Set hostname to . [ 3.614127] random: systemd: uninitialized urandom read (16 bytes read) [ 3.616242] systemd[1]: Initializing machine ID from random generator. [ 3.663769] random: ln: uninitialized urandom read (6 bytes read) [ 3.763161] random: systemd: uninitialized urandom read (16 bytes read) [ 3.765860] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.769965] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.782144] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.449717] device-mapper: uevent: version 1.0.3 [ 4.452447] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.106069] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.160325] virtio_net virtio0 ens2: renamed from eth0 [ 5.310030] scsi host0: ata_piix [ 5.315664] scsi host1: ata_piix [ 5.317440] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.319366] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.070276] dracut-initqueue[581]: RTNETLINK answers: File exists [ 10.013374] random: crng init done [ 10.014982] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.524992] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.714536] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.961295] SELinux: Disabled at runtime. [ 12.016673] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.026659] systemd[1]: Detected virtualization kvm. [ 12.028870] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.543942] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.547564] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.555735] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.560529] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.564737] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.571575] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.580264] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ 12.744034] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.085484] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.514629] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.537735] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.614466] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.646078] EDAC sbridge: Ver: 1.1.2 [ 14.872397] Key type dns_resolver registered [ 15.175467] NFS: Registering the id_resolver key type [ 15.177612] Key type id_resolver registered [ 15.179060] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-client login: [ 65.535621] libcfs: loading out-of-tree module taints kernel. [ 65.638663] Key type ._llcrypt registered [ 65.648782] Key type .llcrypt registered [ 66.432483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 66.442800] alg: No test for adler32 (adler32-zlib) [ 67.755782] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 68.663709] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 70.464299] Key type lgssc registered [ 72.270451] Lustre: Echo OBD driver; http://www.lustre.org/ [ 142.324892] hrtimer: interrupt took 6874608 ns [ 186.218362] Lustre: Mounted lustre-client [ 191.688120] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.506762] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.939795] Lustre: lustre-OST0000-osc-ffff90f2c3b83000: disconnect after 23s idle [ 215.277797] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing yml_node [ 219.055161] Lustre: DEBUG MARKER: Client: 2.16.61.47 [ 221.183892] Lustre: DEBUG MARKER: MDS: 2.16.61.47 [ 223.376569] Lustre: DEBUG MARKER: OSS: 2.16.61.47 [ 224.909590] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 20:19:29 EST 2025 [ 237.564035] LustreError: 7944:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff90f2c3b83000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 237.575428] LustreError: 7944:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 237.600954] Lustre: Unmounted lustre-client [ 254.772750] Key type lgssc unregistered [ 255.116940] LNet: 8390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 255.128940] LNetError: 8390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 255.149211] LNet: Removed LNI 192.168.202.35@tcp [ 255.827876] Key type .llcrypt unregistered [ 255.829235] Key type ._llcrypt unregistered [ 282.104160] Key type ._llcrypt registered [ 282.113958] Key type .llcrypt registered [ 282.453124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 282.493392] alg: No test for adler32 (adler32-zlib) [ 283.662563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 283.829508] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 283.870549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 288.539816] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 288.725345] LNet: 9264:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.729824] LNetError: 9264:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.739974] LNet: Removed LNI 192.168.202.35@tcp [ 289.484405] Key type .llcrypt unregistered [ 289.487676] Key type ._llcrypt unregistered [ 296.591125] Key type ._llcrypt registered [ 296.592754] Key type .llcrypt registered [ 298.030642] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:20:42 (1763342442) [ 298.523909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.555213] alg: No test for adler32 (adler32-zlib) [ 299.457233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.542903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.581571] LNet: 9671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 299.602961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.657713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 299.673957] LNet: 9678:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 307.238935] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:20:51 (1763342451) [ 307.484154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.531635] LNet: 10252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.568460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.698970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 314.188578] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:20:58 (1763342458) [ 314.392074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.417607] LNet: 10841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 314.449948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.521539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 321.397912] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:21:06 (1763342466) [ 321.474711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.525231] LNet: 11417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 321.542237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.678777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 328.988742] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:21:13 (1763342473) [ 329.523718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.562400] LNet: 12009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.618433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.723407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 330.168548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 338.060729] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:21:22 (1763342482) [ 338.185625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.248581] LNet: 12593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.318368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.497845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.206817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 340.153933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 349.466541] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:21:33 (1763342493) [ 349.589950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.660181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.780307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 358.052779] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:21:42 (1763342502) [ 358.116965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.136686] LNet: 13777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.138721] LNet: 13777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 358.156904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.228428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 358.270974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 358.942850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 359.015664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 359.552341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 359.588492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 360.162219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 360.208696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 360.595915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 360.657172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 361.377971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 361.412242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 361.931867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 361.971871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 362.486618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 362.550566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.121224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 363.142723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 363.776986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 363.813580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.283364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 364.347591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 364.800336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 364.856643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 365.300318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 365.332776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 372.253911] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:21:56 (1763342516) [ 372.346932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.371639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.452378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 372.506445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 379.341151] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:22:03 (1763342523) [ 379.408195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.448757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.556484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 379.604588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 386.602297] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:22:10 (1763342530) [ 386.658920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.725557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.832668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 386.884920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 394.570696] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:22:18 (1763342538) [ 394.852576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.899601] LNet: 16225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.909662] LNet: 16225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 394.943732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.023769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 395.061863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 402.846383] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:22:27 (1763342547) [ 402.933760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.976537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.109648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 403.162575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 410.183036] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:22:34 (1763342554) [ 410.439696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.478704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 410.553553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 410.617652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 416.761821] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:22:41 (1763342561) [ 417.200878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.298333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.382679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 417.805151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 424.479703] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:22:49 (1763342569) [ 424.589590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.646922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.904738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 425.217955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 431.693909] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:22:56 (1763342576) [ 431.836259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.896949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.038566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.279147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.569151] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:23:02 (1763342582) [ 438.655501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.707764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.281799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.572966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.133278] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:23:10 (1763342590) [ 446.251657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.341319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.479320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 446.565287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.637265] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:23:18 (1763342598) [ 453.805226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.896523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.069312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.115467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 461.972690] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:23:26 (1763342606) [ 462.109646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.144540] LNet: 21667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 462.154181] LNet: 21667:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 462.193329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.278049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 469.273535] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:23:33 (1763342613) [ 469.419190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.472401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.617546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 476.601628] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:23:41 (1763342621) [ 476.808154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.860085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.972027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 477.387531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 483.319067] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:23:48 (1763342628) [ 483.377171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.406678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.483428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 483.532676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 490.166166] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:23:54 (1763342634) [ 490.295670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.382818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.547338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 490.580886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 498.621426] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:24:02 (1763342642) [ 498.776708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.853638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.992145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 499.048487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 506.626690] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:24:11 (1763342651) [ 506.738176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.818106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.918334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 506.981802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 513.363539] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:24:18 (1763342658) [ 513.460822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.515383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.567313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 513.634968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 513.655890] LNetError: 25786:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 513.699120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 513.739694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 519.268190] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:24:24 (1763342664) [ 519.645210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.686939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.812626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Rx6qyZwPj0 [ 519.874908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Rx6qyZwPj0 [ 519.930453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Rx6qyZwPj0 [ 525.773376] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:24:30 (1763342670) [ 525.856213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.909430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.028990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 526.090706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 526.154191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 532.804272] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:24:37 (1763342677) [ 532.894446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.960169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.053382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 535.771062] LNet: Message forwarding will be enabled in 2s [ 538.848141] LNet: Message forwarding enabled [ 539.228910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 539.251681] LNet: Message forwarding will stop in 2s to 6s [ 543.721673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 544.419946] LNet: Message forwarding disabled [ 545.944512] LNet: Message forwarding will be enabled in 2s [ 549.600118] LNet: Message forwarding enabled [ 553.024619] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:24:57 (1763342697) [ 553.110764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.189939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.251246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 561.323336] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:25:05 (1763342705) [ 561.502919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.609667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.671404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 561.697992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 561.723882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 561.749717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 561.781527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 561.842796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 561.868275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 561.935960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 561.992627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 562.045111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 562.077720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 562.107784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 562.144429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 562.182824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 562.235695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 562.300970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 562.382394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 562.461673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 562.544546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 562.597479] LNetError: 28857:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 570.031588] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:25:14 (1763342714) [ 570.364394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.444872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.534901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 577.721335] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:25:22 (1763342722) [ 579.690233] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg235-server [ 581.741138] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:25:26 (1763342726) [ 581.930377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.981264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 582.025299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 582.079806] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 583.766665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[36-40]@tcp [ 583.796822] LNet: 30574:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 583.889089] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.39@tcp is being used as a gateway but routing feature is not turned on [ 583.916947] LNetError: 30363:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.39@tcp has gone from up to down [ 584.877404] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 584.896613] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 585.894037] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.37@tcp is being used as a gateway but routing feature is not turned on [ 585.924428] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 587.963186] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.37@tcp is being used as a gateway but routing feature is not turned on [ 587.966895] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 590.244664] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:25:35 (1763342735) [ 592.041540] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.40@tcp is being used as a gateway but routing feature is not turned on [ 592.064752] LNetError: 30363:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 592.417648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg235-server [ 594.123674] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:25:38 (1763342738) [ 594.470200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.506331] LNet: 31509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.515341] LNet: 31509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 594.523758] LNetError: 31509:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.571727] LNet: Removed LNI 192.168.202.35@tcp [ 594.618808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.708390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.754825] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 595.653167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 595.675917] LNet: 31626:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 595.703690] LNet: 31626:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 595.716474] LNetError: 31513:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.36@tcp has gone from up to down [ 595.730913] LNetError: 31513:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 595.740327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 602.768972] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:25:47 (1763342747) [ 602.898282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.925644] LNetError: 32212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.945763] LNet: Removed LNI 192.168.202.35@tcp [ 602.975271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.059130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 603.114334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.139536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.216623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.261588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.296719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.344581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 603.414947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.445884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.497411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.570581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.608596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.665533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 603.736297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.770717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.824295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 603.909544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.946888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.005341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 604.078535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.127160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.198132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.265593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.299285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.345576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 604.416153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.453673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.506834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 604.587689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.623113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.666941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 604.723861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.750347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.844547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.344842] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:25:55 (1763342755) [ 612.022054] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg235-server [ 613.793095] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:25:58 (1763342758) [ 615.854779] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg235-server [ 617.957230] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:26:02 (1763342762) [ 618.955234] Key type .llcrypt unregistered [ 618.957312] Key type ._llcrypt unregistered [ 624.425356] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 624.578586] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 624.586157] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 624.595154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 626.178777] Key type ._llcrypt registered [ 626.181638] Key type .llcrypt registered [ 626.419530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 626.436926] alg: No test for adler32 (adler32-zlib) [ 627.353608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.414215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.454640] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 628.145410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 628.161420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 630.151843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 630.185596] LNet: Removed LNI 10.1.2.3@tcp [ 630.313918] LNet: 34322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.321469] LNetError: 34322:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.340631] LNet: Removed LNI 192.168.202.35@tcp [ 631.063676] Key type .llcrypt unregistered [ 631.069874] Key type ._llcrypt unregistered [ 641.934949] Key type ._llcrypt registered [ 641.943122] Key type .llcrypt registered [ 643.518525] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:26:27 (1763342787) [ 644.181787] Key type .llcrypt unregistered [ 644.188787] Key type ._llcrypt unregistered [ 649.751895] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 649.880279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 649.883589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 649.886722] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 652.329451] Key type ._llcrypt registered [ 652.331480] Key type .llcrypt registered [ 652.508434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 652.518308] alg: No test for adler32 (adler32-zlib) [ 653.455585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.502357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 653.527203] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 654.276267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 654.312159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 656.515411] LNet: 35806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.518978] LNetError: 35806:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.521307] LNet: Removed LNI 192.168.202.35@tcp [ 656.536303] LNet: Removed LNI 10.1.2.3@tcp [ 657.145401] Key type .llcrypt unregistered [ 657.147399] Key type ._llcrypt unregistered [ 667.002163] Key type ._llcrypt registered [ 667.007902] Key type .llcrypt registered [ 668.385745] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:26:52 (1763342812) [ 668.960568] Key type .llcrypt unregistered [ 668.962975] Key type ._llcrypt unregistered [ 675.765481] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 676.064548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 676.071390] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 676.073790] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 678.827477] Key type ._llcrypt registered [ 678.828989] Key type .llcrypt registered [ 679.036782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 679.052824] alg: No test for adler32 (adler32-zlib) [ 680.126657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.277104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 680.299980] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 682.739674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 682.773764] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 685.210273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 685.243630] LNet: Removed LNI 10.1.2.3@tcp [ 685.257980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 685.333986] LNet: Removed LNI 10.1.2.31@tcp [ 685.372356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 685.401375] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.230870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 686.285248] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 687.300377] LNet: 37717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.304372] LNetError: 37717:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 687.308092] LNet: Removed LNI 10.1.2.3@tcp [ 687.315510] LNet: Removed LNI 10.1.2.31@tcp [ 687.971305] Key type .llcrypt unregistered [ 687.975831] Key type ._llcrypt unregistered [ 699.280747] Key type ._llcrypt registered [ 699.282420] Key type .llcrypt registered [ 700.510123] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:27:25 (1763342845) [ 700.801915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 700.813884] alg: No test for adler32 (adler32-zlib) [ 701.695631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.749399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.774330] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 702.490340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.506711] LNet: 38666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.512538] LNetError: 38666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.529228] LNet: Removed LNI 192.168.202.35@tcp [ 702.546663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.614584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.35@tcp [ 702.647199] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 703.393504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.35@tcp [ 703.422346] LNet: Removed LNI 192.168.202.35@tcp [ 703.473069] LNet: 38801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.041417] Key type .llcrypt unregistered [ 704.043208] Key type ._llcrypt unregistered [ 714.034549] Key type ._llcrypt registered [ 714.036180] Key type .llcrypt registered [ 715.146928] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:27:39 (1763342859) [ 717.127916] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg235-server [ 718.811896] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:27:43 (1763342863) [ 719.281326] Key type .llcrypt unregistered [ 719.283501] Key type ._llcrypt unregistered [ 724.393702] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 724.559494] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 724.567920] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 724.570662] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 727.055557] Key type ._llcrypt registered [ 727.056928] Key type .llcrypt registered [ 727.291107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 727.315127] alg: No test for adler32 (adler32-zlib) [ 728.214307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.265035] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 728.860202] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 730.796498] LNet: 40630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.808723] LNetError: 40630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 730.820062] LNet: Removed LNI 192.168.202.35@tcp1 [ 730.829738] LNet: Removed LNI 10.1.2.3@tcp2 [ 731.406405] Key type .llcrypt unregistered [ 731.417250] Key type ._llcrypt unregistered [ 743.252625] Key type ._llcrypt registered [ 743.253812] Key type .llcrypt registered [ 744.823603] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:28:09 (1763342889) [ 745.230390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 745.255407] alg: No test for adler32 (adler32-zlib) [ 746.192572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.257452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 746.328411] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 747.309312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.329802] LNet: 41596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.334182] LNetError: 41596:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.343824] LNet: Removed LNI 192.168.202.35@tcp [ 747.367847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.414545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 747.469712] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 754.974717] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:28:19 (1763342899) [ 755.157214] LNet: 42185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.162912] LNetError: 42185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.183300] LNet: Removed LNI 192.168.202.35@tcp [ 755.713619] Key type .llcrypt unregistered [ 755.715377] Key type ._llcrypt unregistered [ 761.415247] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 761.443883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.446216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 761.457687] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.176387] Key type ._llcrypt registered [ 764.178901] Key type .llcrypt registered [ 764.438229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.461947] alg: No test for adler32 (adler32-zlib) [ 765.457317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.542558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 765.583879] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 766.391281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 766.406283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 768.639084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.661555] LNet: 42921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.665777] LNetError: 42921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.672650] LNet: Removed LNI 192.168.202.35@tcp [ 768.690721] LNet: Removed LNI 10.1.2.3@tcp [ 768.706563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.776625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 768.807616] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 768.812974] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 769.532959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.548374] LNet: 42948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.554181] LNetError: 42948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.565166] LNetError: 42948:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 769.576065] LNet: Removed LNI 192.168.202.35@tcp [ 769.586746] LNet: Removed LNI 10.1.2.3@tcp [ 769.616958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.688529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 769.735098] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 769.741401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 770.127687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 770.138508] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 771.167536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.199533] LNet: 43110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.217487] LNetError: 43110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.234226] LNetError: 43110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 771.250481] LNet: Removed LNI 192.168.202.35@tcp [ 771.252305] LNet: Removed LNI 10.1.2.3@tcp [ 771.263239] LNet: Removed LNI 192.168.202.35@tcp2 [ 771.295763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.380946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 771.442868] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 771.454103] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 771.462708] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 780.052493] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:28:44 (1763342924) [ 780.220531] LNet: 43700:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.226705] LNetError: 43700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 780.238956] LNetError: 43700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 780.247280] LNet: Removed LNI 192.168.202.35@tcp [ 780.251354] LNet: Removed LNI 10.1.2.3@tcp [ 780.265786] LNet: Removed LNI 192.168.202.35@tcp2 [ 781.234285] Key type .llcrypt unregistered [ 781.252276] Key type ._llcrypt unregistered [ 787.901719] Key type ._llcrypt registered [ 787.909207] Key type .llcrypt registered [ 788.191709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.206866] alg: No test for adler32 (adler32-zlib) [ 789.216320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.303266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.351734] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 790.572188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 790.606158] LNet: 44099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.611916] LNetError: 44099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.625692] LNet: Removed LNI 192.168.202.35@tcp [ 790.649576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 790.786890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 790.870503] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 791.638993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 791.690397] LNet: 44129:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.717934] LNetError: 44129:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.748795] LNet: Removed LNI 192.168.202.35@tcp [ 791.781902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.941779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 792.006022] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 792.549432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.585795] LNet: Removed LNI 192.168.202.35@tcp [ 792.614528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.681296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 792.711862] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 793.392153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 793.552643] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 793.554818] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 793.557378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 794.987679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.046287] LNet: 44226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.049991] LNet: 44226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 795.055253] LNetError: 44226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 795.058277] LNetError: 44226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 795.068092] LNet: Removed LNI 192.168.202.35@tcp [ 795.086478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.171900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 795.213028] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 802.203657] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:29:06 (1763342946) [ 802.359481] LNet: 44817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.385776] LNetError: 44817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.416701] LNet: Removed LNI 192.168.202.35@tcp [ 803.211739] Key type .llcrypt unregistered [ 803.213562] Key type ._llcrypt unregistered [ 808.646919] Key type ._llcrypt registered [ 808.651283] Key type .llcrypt registered [ 808.885264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 808.907208] alg: No test for adler32 (adler32-zlib) [ 809.843500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.892493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.913315] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 810.772073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 810.792907] LNet: 45218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.801095] LNetError: 45218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.813728] LNet: Removed LNI 192.168.202.35@tcp [ 810.833452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 810.919498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 810.938481] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 817.867032] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:29:22 (1763342962) [ 817.923283] LNet: 45801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 817.937794] LNetError: 45801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 817.957414] LNet: Removed LNI 192.168.202.35@tcp [ 818.715537] Key type .llcrypt unregistered [ 818.720569] Key type ._llcrypt unregistered [ 825.555148] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 825.746510] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 825.754379] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 825.765389] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.278657] Key type ._llcrypt registered [ 828.284272] Key type .llcrypt registered [ 828.479919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.493049] alg: No test for adler32 (adler32-zlib) [ 829.433535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.501510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.564742] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 830.312464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 830.328822] LNet: 46257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.336676] LNetError: 46257:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 830.349609] LNet: Removed LNI 192.168.202.35@tcp [ 830.373750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.425958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 830.453118] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 837.015359] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:29:41 (1763342981) [ 837.272359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 837.289869] LNet: 46855:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.294421] LNetError: 46855:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.310556] LNet: Removed LNI 192.168.202.35@tcp [ 837.338408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 837.461597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 845.584695] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:29:49 (1763342989) [ 845.857915] LNet: 47437:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.812680] Key type .llcrypt unregistered [ 846.817767] Key type ._llcrypt unregistered [ 854.073123] Key type ._llcrypt registered [ 854.074824] Key type .llcrypt registered [ 854.368659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 854.400302] alg: No test for adler32 (adler32-zlib) [ 855.342828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.406468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 855.430803] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 856.464116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 856.483840] LNet: 47832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.496477] LNetError: 47832:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 856.518757] LNet: Removed LNI 192.168.202.35@tcp [ 856.546611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.646099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 856.699042] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 863.239831] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:30:07 (1763343007) [ 863.381257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.405589] LNet: 48423:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.411702] LNetError: 48423:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.450259] LNet: Removed LNI 192.168.202.35@tcp [ 863.487545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.561259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 869.673348] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:30:14 (1763343014) [ 871.007790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 872.980979] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:30:17 (1763343017) [ 874.900142] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 876.313981] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:30:21 (1763343021) [ 877.806865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 879.641667] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:30:24 (1763343024) [ 881.086352] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 883.003660] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:30:27 (1763343027) [ 884.622523] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 886.342909] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:30:31 (1763343031) [ 886.574760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.613626] LNet: 50477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.751947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.813115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 886.858259] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 887.727965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.35 [ 887.753777] LNet: Removed LNI 192.168.202.35@tcp [ 894.084977] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:30:38 (1763343038) [ 894.253118] LNet: 51174:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.271234] LNetError: 51174:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.312534] LNet: Removed LNI 192.168.202.35@tcp [ 894.994966] Key type .llcrypt unregistered [ 894.996695] Key type ._llcrypt unregistered [ 901.031229] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 901.318946] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 901.321854] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 901.326066] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.003211] Key type ._llcrypt registered [ 904.004769] Key type .llcrypt registered [ 904.246964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.267774] alg: No test for adler32 (adler32-zlib) [ 905.658081] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 912.625242] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:30:57 (1763343057) [ 912.853974] LNet: 52080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.861397] LNetError: 52080:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.894400] LNet: Removed LNI 10.1.2.3@tcp [ 913.653599] Key type .llcrypt unregistered [ 913.655722] Key type ._llcrypt unregistered [ 919.020342] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 919.288543] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 919.291063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 919.293678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 921.967345] Key type ._llcrypt registered [ 921.968943] Key type .llcrypt registered [ 922.108343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 922.124445] alg: No test for adler32 (adler32-zlib) [ 923.082957] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 930.045180] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:31:14 (1763343074) [ 930.482656] LNet: 52973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 930.500767] LNetError: 52973:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 930.517212] LNet: Removed LNI 10.1.2.3@tcp [ 931.113669] Key type .llcrypt unregistered [ 931.115803] Key type ._llcrypt unregistered [ 936.632527] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 936.932825] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 936.943194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 936.946878] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 939.792911] Key type ._llcrypt registered [ 939.795197] Key type .llcrypt registered [ 940.036225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.046365] alg: No test for adler32 (adler32-zlib) [ 941.008636] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 946.944693] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:31:31 (1763343091) [ 947.176394] LNet: 53861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.185802] LNetError: 53861:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.206991] LNet: Removed LNI 10.1.2.3@tcp [ 947.749480] Key type .llcrypt unregistered [ 947.756163] Key type ._llcrypt unregistered [ 952.992024] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 953.172618] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 953.176678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 953.182873] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 955.819786] Key type ._llcrypt registered [ 955.821192] Key type .llcrypt registered [ 955.961493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.972872] alg: No test for adler32 (adler32-zlib) [ 956.900026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.968050] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 965.149136] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:31:49 (1763343109) [ 965.710094] LNet: 55086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.722493] LNetError: 55086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 965.749955] LNet: Removed LNI 10.1.2.3@tcp [ 966.637724] Key type .llcrypt unregistered [ 966.639298] Key type ._llcrypt unregistered [ 972.416049] Key type ._llcrypt registered [ 972.417276] Key type .llcrypt registered [ 972.659292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 972.685288] alg: No test for adler32 (adler32-zlib) [ 973.645623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 973.671800] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 979.949363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 980.932278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 980.977462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 981.021404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 981.057302] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 981.294936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 981.327950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 981.357403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 981.396411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 981.435249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 981.458946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 981.495828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 981.534166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 981.552768] LNet: 55520:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 981.575481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 981.627338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 981.658741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 981.690068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 981.720593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 981.752700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 981.779146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 982.241890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 982.278659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 982.311026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 982.329483] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 982.523665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 982.561832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 982.597270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 982.645240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 982.683293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 982.706307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 982.736951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 982.769604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 982.785542] LNet: 55629:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 982.800665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 982.827253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 982.848877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 982.870748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 982.880351] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 982.917320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 982.959853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 983.002716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 983.468816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 983.493048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 983.523636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 983.538554] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 983.736626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 983.766406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 983.789115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 983.828095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 983.854984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 983.890518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 983.922135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 983.943276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 983.962623] LNet: 55739:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 983.973614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 984.004665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 984.037742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 984.086176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 984.113515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 984.154883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 984.183410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 984.616898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 984.680672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 984.705590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 984.876312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 984.902569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 984.937231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 984.971613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 985.009692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 985.041135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 985.071570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 985.094674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 985.121470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 985.150385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 985.181758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 985.209581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 985.262912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 985.287206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 985.327759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 985.734635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 985.758096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 985.801128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 985.827224] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 985.831875] LNet: Skipped 2 previous similar messages [ 986.039417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 986.069704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 986.097827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 986.129016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 986.155104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 986.183576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 986.210915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 986.244472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 986.261951] LNet: 55957:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 986.265970] LNet: 55957:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 986.276834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 986.335514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 986.367653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 986.394150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 986.421886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 986.466177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 986.532544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 987.064961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 987.091876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 987.124854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 987.350572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 987.387282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 987.419386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 987.443445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 987.470376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 987.494826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 987.524882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 987.551909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 987.578775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 987.607273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 987.633941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 987.656976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 987.685697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 987.714265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 987.743813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 987.998990] LNet: 56106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.014533] LNetError: 56106:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 988.044361] LNet: Removed LNI 192.168.202.35@tcp [ 988.471367] Key type .llcrypt unregistered [ 988.488229] Key type ._llcrypt unregistered [ 1002.504857] Key type ._llcrypt registered [ 1002.506485] Key type .llcrypt registered [ 1003.655136] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:32:28 (1763343148) [ 1004.375377] Key type .llcrypt unregistered [ 1004.377916] Key type ._llcrypt unregistered [ 1009.532351] Key type ._llcrypt registered [ 1009.534024] Key type .llcrypt registered [ 1009.741929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.756720] alg: No test for adler32 (adler32-zlib) [ 1010.703582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1010.748820] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1017.623335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1026.528341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1026.586475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1026.607799] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1028.053224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1028.117699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 1028.168555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1028.204525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 1028.256236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 1028.292780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 1028.320667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 1028.350898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 1028.383419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1028.399315] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1028.405938] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1028.648293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.696442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1028.720258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.746724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.777509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.824939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.895183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.934466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.950233] LNet: 57567:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.963875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.015148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.048597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.096106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.147228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.199262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.236894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.745700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1029.810710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 1029.881565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1029.965734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 1030.025399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 1030.107330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 1030.171302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 1030.219456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 1030.266278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1030.295528] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1030.298935] LNet: Skipped 2 previous similar messages [ 1030.619621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.692113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1030.732317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.773124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.831584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.876059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.907339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.934662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.947877] LNet: 57689:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.966695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.010916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.036456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.073924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.114474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.146432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.177747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1031.780367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1031.812702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 1031.843341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1031.876854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 1031.912408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 1031.945918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 1031.982327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 1032.012492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 1032.055597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1032.088608] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1032.095751] LNet: Skipped 2 previous similar messages [ 1032.415289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.441784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1032.470879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.502757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.524912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.553420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.576475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.609253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.626244] LNet: 57810:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1032.638477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.671382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.709262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.730442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.763396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.785934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.820399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1033.293104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1033.336049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 1033.365796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1033.401558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 1033.427160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 1033.457094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 1033.497513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 1033.521829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 1033.555341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1033.782092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1033.840500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1033.871516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1033.894961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1033.921751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1033.975524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1034.027550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1034.055833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1034.081752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1034.125762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1034.161157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1034.184895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1034.212786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1034.252817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1034.279180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1034.792554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1034.826273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 1034.861433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1034.887424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 1034.913715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 1034.945661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 1034.977888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 1035.020675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 1035.056342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1035.087681] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1035.097519] LNet: Skipped 5 previous similar messages [ 1035.300431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.354984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.397373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.438161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1035.471694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1035.507567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1035.561351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1035.587869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1035.600979] LNet: 58053:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1035.611019] LNet: 58053:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1035.631273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1035.679518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1035.743789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1035.783109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1035.836851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.870323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1035.893099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1036.365061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1036.424875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 1036.487852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1036.544798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 1036.589864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 1036.638391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 1036.685580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 1036.732431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 1036.785548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1037.055637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1037.096460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1037.127121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1037.174767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1037.227431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1037.257491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1037.285451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1037.315816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1037.338269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1037.369313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1037.399296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1037.437600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1037.477600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1037.519287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1037.588022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1042.291855] LNet: 58249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.302463] LNetError: 58249:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1042.307930] LNet: Removed LNI 192.168.202.35@tcp [ 1043.366387] LNet: Removed LNI 192.168.202.35@tcp1 [ 1043.743676] Key type .llcrypt unregistered [ 1043.745053] Key type ._llcrypt unregistered [ 1058.536208] Key type ._llcrypt registered [ 1058.538988] Key type .llcrypt registered [ 1059.741319] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:33:24 (1763343204) [ 1060.506257] Key type .llcrypt unregistered [ 1060.509627] Key type ._llcrypt unregistered [ 1066.098111] Key type ._llcrypt registered [ 1066.099689] Key type .llcrypt registered [ 1066.330665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.349624] alg: No test for adler32 (adler32-zlib) [ 1067.353553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1067.386264] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1074.287598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1075.004992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1075.041784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1075.082978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1075.100589] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1075.337324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.389124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1075.424108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.470760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.502707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.543074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.596796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.634948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.663940] LNet: 59493:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.703929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.756385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.801143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.843449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.908841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.939120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.982122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.527466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1076.562164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1076.589569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1076.607514] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1076.825282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.851497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1076.883790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.917365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.949162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.978815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.999360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1077.014224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.023807] LNet: 59603:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1077.032775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.047742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.095362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.138505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.176695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.206373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.244497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1077.776475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1077.846771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1077.895165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1077.917184] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1078.138592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1078.182816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1078.222165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1078.287765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1078.340771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1078.381889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1078.428234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1078.478041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1078.571458] LNet: 59712:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1078.605956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1078.662076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1078.710761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1078.766530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1078.818567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1078.865330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1078.912791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1079.146611] LNet: 59752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.152978] LNetError: 59752:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.173230] LNet: Removed LNI 192.168.202.35@tcp [ 1080.575792] Key type .llcrypt unregistered [ 1080.579899] Key type ._llcrypt unregistered [ 1096.120785] Key type ._llcrypt registered [ 1096.122742] Key type .llcrypt registered [ 1097.523507] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:34:01 (1763343241) [ 1098.709560] Key type .llcrypt unregistered [ 1098.713844] Key type ._llcrypt unregistered [ 1104.770746] Key type ._llcrypt registered [ 1104.772462] Key type .llcrypt registered [ 1104.956120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.972147] alg: No test for adler32 (adler32-zlib) [ 1105.927845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1105.965053] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1113.332776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1121.724712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.773764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1121.821710] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1123.672034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1123.749482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1123.812741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1123.856121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1123.898402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1123.929130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1123.956682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1123.994873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1124.019703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1124.036374] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1124.040241] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1124.307657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.344941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1124.374284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.413341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.438426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.458139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.503180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.544243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.590521] LNet: 61216:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.613096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.657454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.685449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.717568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.740863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.764773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.795152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.823635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.374509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1125.435465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1125.485930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1125.558822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1125.635265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1125.686616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1125.718330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1125.762515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1125.815628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1125.861780] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1125.873779] LNet: Skipped 2 previous similar messages [ 1126.168435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.193410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1126.231477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.255971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.280870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.306458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.332291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.364551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1126.379689] LNet: 61339:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1126.399175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1126.435250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1126.459784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1126.486078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1126.502586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1126.536328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1126.571682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1126.596510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.073762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1127.143667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1127.162432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1127.200947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1127.236442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1127.262684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1127.294912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1127.350163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1127.384854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1127.422742] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1127.427653] LNet: Skipped 1 previous similar message [ 1127.610753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1127.631798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1127.666811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1127.691602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1127.722639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1127.751705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1127.780218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1127.826189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.837981] LNet: 61463:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1127.879378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.919096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.948242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.972585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.016613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.058862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.095366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1128.125707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.155526] LNet: 61540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.166081] LNetError: 61540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.216517] LNet: Removed LNI 192.168.202.35@tcp [ 1133.218657] LNetError: 61540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.238521] LNet: Removed LNI 192.168.202.35@tcp1 [ 1133.686787] Key type .llcrypt unregistered [ 1133.688629] Key type ._llcrypt unregistered [ 1147.013191] Key type ._llcrypt registered [ 1147.015265] Key type .llcrypt registered [ 1148.324351] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:34:52 (1763343292) [ 1149.034362] Key type .llcrypt unregistered [ 1149.035957] Key type ._llcrypt unregistered [ 1154.485206] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1154.687894] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1154.698783] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1154.708358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1157.207347] Key type ._llcrypt registered [ 1157.208824] Key type .llcrypt registered [ 1157.362554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1157.373580] alg: No test for adler32 (adler32-zlib) [ 1158.276064] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1158.310726] LNet: 62619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1158.316421] LNetError: 62619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1158.329699] LNet: Removed LNI 192.168.202.35@tcp [ 1158.902488] Key type .llcrypt unregistered [ 1158.905519] Key type ._llcrypt unregistered [ 1163.659663] Key type ._llcrypt registered [ 1163.663406] Key type .llcrypt registered [ 1163.829218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.864864] alg: No test for adler32 (adler32-zlib) [ 1164.787091] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1164.797066] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1164.855403] LNet: 62888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.868569] LNetError: 62888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.876473] LNet: Removed LNI 192.168.202.35@tcp [ 1164.888468] LNet: Removed LNI 10.1.2.3@tcp1 [ 1165.448850] Key type .llcrypt unregistered [ 1165.450721] Key type ._llcrypt unregistered [ 1170.590385] Key type ._llcrypt registered [ 1170.596542] Key type .llcrypt registered [ 1170.782523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1170.797238] alg: No test for adler32 (adler32-zlib) [ 1171.751942] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1171.758543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1171.794654] LNet: 63157:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1171.802523] LNetError: 63157:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1171.806209] LNet: Removed LNI 192.168.202.35@tcp [ 1171.816229] LNet: Removed LNI 10.1.2.3@tcp [ 1172.434398] Key type .llcrypt unregistered [ 1172.436249] Key type ._llcrypt unregistered [ 1177.816190] Key type ._llcrypt registered [ 1177.817470] Key type .llcrypt registered [ 1178.084858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1178.107340] alg: No test for adler32 (adler32-zlib) [ 1179.097067] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1179.098910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1179.160218] LNet: 63426:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1179.167171] LNetError: 63426:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1179.173743] LNet: Removed LNI 192.168.202.35@tcp [ 1179.186853] LNet: Removed LNI 10.1.2.3@tcp [ 1179.800854] Key type .llcrypt unregistered [ 1179.802640] Key type ._llcrypt unregistered [ 1185.277143] Key type ._llcrypt registered [ 1185.280914] Key type .llcrypt registered [ 1185.401919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1185.411984] alg: No test for adler32 (adler32-zlib) [ 1186.358135] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1186.360094] LNetError: 63685:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1186.365330] LNetError: Error -100 starting up LNI tcp [ 1186.379714] LNet: Removed LNI 192.168.202.35@tcp [ 1186.950639] Key type .llcrypt unregistered [ 1186.956728] Key type ._llcrypt unregistered [ 1196.402988] Key type ._llcrypt registered [ 1196.412053] Key type .llcrypt registered [ 1197.808751] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:35:42 (1763343342) [ 1198.745784] Key type .llcrypt unregistered [ 1198.749302] Key type ._llcrypt unregistered [ 1203.987934] Key type ._llcrypt registered [ 1203.994629] Key type .llcrypt registered [ 1204.242883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1204.292618] alg: No test for adler32 (adler32-zlib) [ 1205.251204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1205.275624] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1211.706449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1212.556385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1212.614546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1212.655515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1212.694524] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1212.937656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.970585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.995429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1213.027796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1213.057453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1213.093534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1213.134240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1213.166957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1213.183566] LNet: 64934:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1213.207396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1213.242801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1213.287025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1213.311436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1213.335024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.364734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1213.666275] LNet: 64972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1213.676528] LNetError: 64972:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.697223] LNet: Removed LNI 192.168.202.35@tcp [ 1215.023466] Key type .llcrypt unregistered [ 1215.026727] Key type ._llcrypt unregistered [ 1231.212327] Key type ._llcrypt registered [ 1231.217507] Key type .llcrypt registered [ 1231.436920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1231.449454] alg: No test for adler32 (adler32-zlib) [ 1232.451848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1232.500808] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1239.529554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1246.699377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.748461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1246.769217] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1248.191240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1248.241617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1248.273439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1248.313200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1248.348739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1248.386390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1248.421777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1248.471694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1248.523092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1248.548167] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1248.758576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1248.789687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1248.819229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.852309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.879593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.913185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.930394] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1248.935739] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1248.975712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.996419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.010454] LNet: 65844:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1249.031877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.059618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.090410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.117883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.143159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.162654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1252.868821] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1253.120281] LNet: 65917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.129216] LNetError: 65917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.136206] LNet: Removed LNI 192.168.202.35@tcp [ 1253.149314] LNet: Removed LNI 192.168.202.35@tcp1 [ 1253.534303] Key type .llcrypt unregistered [ 1253.540388] Key type ._llcrypt unregistered [ 1266.609459] Key type ._llcrypt registered [ 1266.611079] Key type .llcrypt registered [ 1267.991135] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:36:52 (1763343412) [ 1268.255717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1268.274600] alg: No test for adler32 (adler32-zlib) [ 1269.205415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.257451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.285439] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1270.011614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1270.026920] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1270.790423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1270.835410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1270.877573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1270.903424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1270.933852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1270.953545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1270.978906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1270.991687] LNetError: 67013:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1271.264344] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1271.266448] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1273.312436] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1274.450228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1277.408511] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1277.887520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1277.926709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1277.975752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.994252] LNet: 67120:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.004205] LNetError: 67120:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.010768] LNet: Removed LNI 192.168.202.35@tcp [ 1278.019724] LNet: Removed LNI 192.168.202.35@tcp1 [ 1278.038362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.080994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.118163] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1278.819130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1278.836723] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1279.650727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1279.677711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.735191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1279.782596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1279.818853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1279.853208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1279.896982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1279.926639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1279.943745] LNetError: 67392:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1280.097425] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1282.336731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1286.240481] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1286.251373] LNet: Skipped 1 previous similar message [ 1291.350073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1291.384668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1291.420925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1296.924896] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:37:21 (1763343441) [ 1297.221764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.258683] LNet: 68108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1297.273562] LNetError: 68108:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1297.281894] LNetError: 68108:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1297.287599] LNet: Removed LNI 192.168.202.35@tcp [ 1297.316909] LNet: Removed LNI 192.168.202.35@tcp1 [ 1297.344800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.414646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.451248] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1298.171348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1298.198750] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1298.881588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1298.942312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1298.967501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1299.001927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1299.043420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1299.073980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1299.115028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1299.200589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1299.425628] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.35@tcp [ 1300.449662] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1300.453059] LNet: Skipped 1 previous similar message [ 1315.567858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.701716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1315.746000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1315.780934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 500 [ 1315.890108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.574633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.650543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.673134] LNet: 68609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.679934] LNetError: 68609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.686246] LNetError: 68609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1322.691315] LNet: Removed LNI 192.168.202.35@tcp [ 1322.700992] LNet: Removed LNI 192.168.202.35@tcp1 [ 1322.733816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1322.829593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1322.882731] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1323.670817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1323.689426] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1324.534324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1324.571840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1324.640952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1324.674943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1324.704903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1324.732297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1324.764465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1324.802344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1325.858810] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1325.863638] LNet: Skipped 3 previous similar messages [ 1326.096676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.201330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1336.239631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.271342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.835567] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:38:07 (1763343487) [ 1343.161571] LNet: 69653:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.167359] LNetError: 69653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.173158] LNetError: 69653:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1343.176898] LNet: Removed LNI 192.168.202.35@tcp [ 1343.191460] LNet: Removed LNI 192.168.202.35@tcp1 [ 1343.851895] Key type .llcrypt unregistered [ 1343.854844] Key type ._llcrypt unregistered [ 1348.671698] Key type ._llcrypt registered [ 1348.672861] Key type .llcrypt registered [ 1348.834451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1348.855983] alg: No test for adler32 (adler32-zlib) [ 1349.771316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1349.790383] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1356.022067] Lustre: DEBUG MARKER: Initial discovery [ 1356.227140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1358.278824] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1358.403827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1358.443380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1358.475576] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1358.489877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1358.538139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -l 3 [ 1358.589357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -e local_error [ 1358.636742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m PUT -l 6 [ 1359.017062] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1360.026314] Lustre: DEBUG MARKER: Force oleg235-client.virtnet to discover 192.168.202.135@tcp (in background) [ 1360.091437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1361.239327] Lustre: DEBUG MARKER: Force oleg235-server to discover 192.168.202.35@tcp [ 1363.173161] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1363.409028] Lustre: DEBUG MARKER: Wait for 70402 [ 1364.664806] Lustre: DEBUG MARKER: Finished wait on 70402 [ 1364.724089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1364.766938] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1364.778371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1365.057355] LNet: 70860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.065266] LNetError: 70860:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.085983] LNet: Removed LNI 192.168.202.35@tcp [ 1365.410656] Key type .llcrypt unregistered [ 1365.412312] Key type ._llcrypt unregistered [ 1375.184299] Key type ._llcrypt registered [ 1375.186019] Key type .llcrypt registered [ 1376.161537] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:38:41 (1763343521) [ 1376.681312] Key type .llcrypt unregistered [ 1376.683954] Key type ._llcrypt unregistered [ 1380.547329] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1380.666320] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1380.673968] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1380.680635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1383.104342] Key type ._llcrypt registered [ 1383.105986] Key type .llcrypt registered [ 1383.231702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1383.242660] alg: No test for adler32 (adler32-zlib) [ 1384.133545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.179799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1384.199892] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1384.729198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1384.745363] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1391.975123] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:38:56 (1763343536) [ 1392.269624] LNet: 72926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1392.273866] LNetError: 72926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.276949] LNet: Removed LNI 192.168.202.35@tcp [ 1392.291455] LNet: Removed LNI 10.1.2.3@tcp [ 1392.859487] Key type .llcrypt unregistered [ 1392.861712] Key type ._llcrypt unregistered [ 1397.681649] Key type ._llcrypt registered [ 1397.685920] Key type .llcrypt registered [ 1397.869462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1397.877694] alg: No test for adler32 (adler32-zlib) [ 1398.780813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.829980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1398.852070] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1399.392694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1399.422229] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1400.240345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.35@tcp [ 1401.315674] LNetError: Unexpected error -22 connecting to 192.168.202.35@tcp at host 192.168.202.35:988 [ 1406.519770] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:39:11 (1763343551) [ 1406.633492] LNet: 74045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.642015] LNetError: 74045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.650383] LNet: Removed LNI 192.168.202.35@tcp [ 1407.657592] LNet: Removed LNI 10.1.2.3@tcp [ 1408.358462] Key type .llcrypt unregistered [ 1408.360190] Key type ._llcrypt unregistered [ 1413.818272] Key type ._llcrypt registered [ 1413.819932] Key type .llcrypt registered [ 1414.163946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1414.196223] alg: No test for adler32 (adler32-zlib) [ 1415.098127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.150258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1415.189044] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1415.780711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1415.795710] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1416.672292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.35@tcp1 --nid 192.168.202.35@tcp2 [ 1419.672397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp2 [ 1419.689715] LNetError: 74885:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.35@tcp1 to 192.168.202.35@tcp2 - Route table may be misconfigured [ 1419.702874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp2 192.168.202.35@tcp1 [ 1425.065669] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:39:30 (1763343570) [ 1426.219213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.238965] LNet: 75459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.249578] LNetError: 75459:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1426.253752] LNet: Removed LNI 192.168.202.35@tcp1 [ 1427.306169] LNet: Removed LNI 192.168.202.35@tcp2 [ 1427.329109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.382966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1427.414754] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1428.220365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1428.242089] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1429.106883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1429.144283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.35@tcp [ 1429.175784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp1 [ 1429.202554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp -e network_timeout [ 1429.232504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -e network_timeout [ 1429.267528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -e network_timeout [ 1429.303484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -e network_timeout [ 1429.358478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1429.393421] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1429.407869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1429.408137] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1429.499027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1429.597582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1436.436149] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:39:41 (1763343581) [ 1436.599983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.635111] LNet: 76319:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1436.647571] LNetError: 76319:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1436.660025] LNetError: 76319:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1436.665600] LNet: Removed LNI 192.168.202.35@tcp [ 1437.736331] LNet: Removed LNI 192.168.202.35@tcp1 [ 1437.791116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1437.907887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1438.255824] LNet: 76358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.673355] Key type .llcrypt unregistered [ 1438.675112] Key type ._llcrypt unregistered [ 1448.111149] Key type ._llcrypt registered [ 1448.112678] Key type .llcrypt registered [ 1449.135717] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:39:54 (1763343594) [ 1449.576083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1449.587624] alg: No test for adler32 (adler32-zlib) [ 1450.464511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.610309] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1453.271482] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:39:57 (1763343597) [ 1453.516144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.539939] LNet: 77452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.560883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.620172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.642722] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1454.482156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1454.518696] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1455.298612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1455.333365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp1 [ 1455.366252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp1 [ 1460.768930] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:40:05 (1763343605) [ 1462.323707] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg235-server [ 1463.929780] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:40:08 (1763343608) [ 1465.531147] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg235-server [ 1467.482863] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:40:11 (1763343611) [ 1469.403573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg235-server [ 1471.211213] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:40:15 (1763343615) [ 1473.036965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg235-server [ 1475.085929] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:40:19 (1763343619) [ 1476.691552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg235-server [ 1478.254049] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:40:23 (1763343623) [ 1479.962664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg235-server [ 1481.744600] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:40:26 (1763343626) [ 1483.514889] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg235-server [ 1485.803238] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:40:30 (1763343630) [ 1487.585792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg235-server [ 1489.165886] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:40:34 (1763343634) [ 1490.719720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg235-server [ 1492.418584] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:40:37 (1763343637) [ 1492.501535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.527521] LNet: 81323:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.538507] LNetError: 81323:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.544629] LNet: Removed LNI 192.168.202.35@tcp [ 1493.608949] LNet: Removed LNI 192.168.202.35@tcp1 [ 1493.664771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1493.733949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1493.765192] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1494.679561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1494.778939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1495.017955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1495.036709] LNet: 81476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1495.054891] LNetError: 81476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1495.057926] LNetError: 81476:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1496.106334] LNet: Removed LNI 192.168.202.35@tcp [ 1496.154345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.255174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1496.293248] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1497.097681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1497.325862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1497.382614] LNet: 81621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.399986] LNetError: 81621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1497.428794] LNet: Removed LNI 192.168.202.35@tcp [ 1497.459242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1497.519250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1497.593261] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1498.530271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1498.647624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1500.165688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1500.210921] LNetError: 81778:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.254348] LNet: Removed LNI 192.168.202.35@tcp [ 1500.288756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1500.373751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1500.424121] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1501.263892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1501.499767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1501.544108] LNet: 81923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1501.551967] LNet: 81923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1502.631221] LNet: Removed LNI 192.168.202.35@tcp [ 1502.670130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1502.721090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1502.777491] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1503.577506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1503.654867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1503.909571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.003906] LNet: Removed LNI 192.168.202.35@tcp [ 1505.055198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.141653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1505.200175] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1506.239775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1507.645695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1507.684744] LNetError: 82225:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1507.689523] LNetError: 82225:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1507.706899] LNet: Removed LNI 192.168.202.35@tcp [ 1507.747233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.797598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.820932] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1508.563703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1508.648360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1508.913456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.996470] LNet: Removed LNI 192.168.202.35@tcp [ 1510.034193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.060937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1510.087693] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1510.935105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1512.369196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1512.397771] LNet: 82526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1512.407746] LNet: 82526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1513.453672] LNet: Removed LNI 192.168.202.35@tcp [ 1513.483860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.580660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.632874] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1514.423651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1514.542785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1514.937686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.008881] LNet: Removed LNI 192.168.202.35@tcp [ 1516.060645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.166216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1516.214609] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1517.001980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1517.405755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1517.429875] LNetError: 82819:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.433384] LNetError: 82819:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1517.444759] LNet: Removed LNI 192.168.202.35@tcp [ 1517.488118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.574340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.629733] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1518.579401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1518.712311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1519.122782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1520.233661] LNet: Removed LNI 192.168.202.35@tcp [ 1520.267634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1520.340122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1520.406362] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1521.384891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1521.757140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.796447] LNet: Removed LNI 192.168.202.35@tcp [ 1522.871767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.991475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1523.035906] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1523.961196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1524.008829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.040705] LNet: Removed LNI 192.168.202.35@tcp [ 1524.070475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.130221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.158783] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1524.983310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1531.687291] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:41:16 (1763343676) [ 1531.897544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1531.923897] LNet: 83950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.933444] LNet: 83950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1531.953983] LNet: Removed LNI 192.168.202.35@tcp [ 1531.974287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.055871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1532.089380] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1533.464117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.489779] LNetError: 84088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.494632] LNetError: 84088:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1533.511782] LNet: Removed LNI 192.168.202.35@tcp231 [ 1533.533500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.601309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1533.659114] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1533.680058] LNet: 84096:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1534.172485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1534.205044] LNet: Removed LNI 192.168.202.35@tcp231 [ 1534.229649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1534.272142] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1535.393715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1535.427923] LNet: Removed LNI 192.168.202.35@tcp231 [ 1535.461160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1535.559724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1535.583226] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1535.589190] LNet: 84248:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1536.090178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1536.136986] LNet: Removed LNI 192.168.202.35@tcp231 [ 1536.166752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1536.195424] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1537.453384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.507258] LNet: Removed LNI 192.168.202.35@tcp231 [ 1537.530613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1537.601821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1537.639583] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1537.647169] LNet: 84400:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1538.053778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1538.096677] LNet: Removed LNI 192.168.202.35@tcp231 [ 1538.116969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1538.139508] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1539.095848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.146033] LNet: Removed LNI 192.168.202.35@tcp231 [ 1539.173306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.241822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1539.271247] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1539.752536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1539.803524] LNet: Removed LNI 192.168.202.35@tcp231 [ 1539.821874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1539.859746] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1546.787574] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:41:31 (1763343691) [ 1547.013301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.084393] LNet: Removed LNI 192.168.202.35@tcp231 [ 1547.111217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.185971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1547.213308] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1547.941126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1547.983167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1548.004144] LNet: Removed LNI 192.168.202.35@tcp232 [ 1548.021084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1548.040990] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1548.817768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1548.864968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1548.901368] LNet: Removed LNI 192.168.202.35@tcp232 [ 1548.920173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1548.961597] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1549.869903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1549.899095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1549.955542] LNet: Removed LNI 192.168.202.35@tcp232 [ 1549.982923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1550.034622] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1550.805728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1550.844049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1550.877644] LNet: Removed LNI 192.168.202.35@tcp232 [ 1550.891119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1556.358620] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:41:41 (1763343701) [ 1557.340748] Key type .llcrypt unregistered [ 1557.343291] Key type ._llcrypt unregistered [ 1562.239134] Key type ._llcrypt registered [ 1562.242625] Key type .llcrypt registered [ 1562.435710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1562.451247] alg: No test for adler32 (adler32-zlib) [ 1563.380671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1563.431757] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1568.930483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1575.121050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.160665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1575.180290] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1576.414394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 1576.446903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 1576.479473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 1576.520711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 1576.555788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 1576.590787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 1576.627962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 1576.658625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 1576.724988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1576.738051] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1576.741141] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1576.968244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1576.986052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1577.006069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1577.032080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1577.054533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1577.089639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1577.129891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1577.161852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1577.191357] LNet: 87046:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.211899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.249538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.281441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1577.304873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1577.330936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1577.358330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1577.912737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 1577.952734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 1577.984719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 1578.010038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 1578.045425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 1578.080142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 1578.111220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 1578.145301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 1578.185414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1578.206313] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1578.440465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.473772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1578.508525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.540127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.574450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1578.603739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1578.623378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.653127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1578.666482] LNet: 87167:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1578.686482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1578.706566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1578.733512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.766515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1578.802304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1578.847713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1579.508662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 1579.585270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 1579.660852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 1579.701648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 1579.754515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 1579.786567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 1579.841412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 1579.879700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 1579.919912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1579.957905] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1580.211123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1580.243217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1580.274356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1580.292239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1580.355388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1580.398388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1580.451559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1580.510513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1580.532969] LNet: 87287:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1580.558774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1580.608569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1580.666874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1580.693569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1580.744457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1580.784432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1581.323903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 1581.358687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 1581.393091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 1581.424971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 1581.464336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 1581.498353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 1581.549768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 1581.585800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 1581.621936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1581.941453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1581.987367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1582.070489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1582.120483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1582.155951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1582.202859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1582.244072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1582.283421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1582.306421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1582.327718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1582.394412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1582.436288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1582.466572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1582.505360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1582.919587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 1582.943960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 1582.979690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 1582.996382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 1583.011799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 1583.048161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 1583.125400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 1583.149574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 1583.184141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1583.197263] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1583.200284] LNet: Skipped 1 previous similar message [ 1583.472215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1583.522342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1583.561561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1583.617259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1583.653700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1583.696540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1583.742661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1583.787124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1583.806572] LNet: 87528:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1583.812331] LNet: 87528:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1583.831937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1583.870335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1583.931283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1583.965695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1583.994630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1584.037987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1584.612117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 1584.652915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 1584.689351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 1584.725851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 1584.758633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 1584.796041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 1584.826121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 1584.861437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 1584.896273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1584.918603] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1585.141996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1585.185840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1585.222723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1585.270387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1585.295176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1585.318181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1585.351916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1585.378073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1585.400484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1585.448427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1585.512797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1585.542039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1585.600914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1585.645964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1590.482546] LNet: 87721:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.492800] LNetError: 87721:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.501922] LNet: Removed LNI 192.168.202.35@tcp [ 1590.517959] LNet: Removed LNI 192.168.202.35@tcp1 [ 1590.812518] Key type .llcrypt unregistered [ 1590.815384] Key type ._llcrypt unregistered [ 1606.311396] Key type ._llcrypt registered [ 1606.312811] Key type .llcrypt registered [ 1607.630907] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:42:32 (1763343752) [ 1608.057718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.087850] alg: No test for adler32 (adler32-zlib) [ 1609.058395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1609.098966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1609.119213] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1610.024697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1610.112888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1610.171839] LNetError: 88697:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1610.413175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.35@tcp [ 1610.570255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.780607] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:42:42 (1763343762) [ 1617.884189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.938434] LNet: 89280:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.958602] LNetError: 89280:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1617.998950] LNet: Removed LNI 192.168.202.35@tcp [ 1618.025122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1618.090166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1618.117563] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1619.080139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1619.112516] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1620.763520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.35@tcp [ 1620.800180] LNetError: 89553:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1621.155915] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1621.164057] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1622.127072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.35@tcp [ 1630.839757] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:42:55 (1763343775) [ 1632.917593] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg235-server [ 1634.970500] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:42:59 (1763343779) [ 1635.077173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.113377] LNet: 90495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.125321] LNetError: 90495:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.128682] LNet: Removed LNI 192.168.202.35@tcp [ 1635.150307] LNet: Removed LNI 192.168.202.35@tcp1 [ 1635.185298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.251953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.281928] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1636.139456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1636.228658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1636.286424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1636.368371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1636.482166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1636.570602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1645.016453] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:43:09 (1763343789) [ 1645.112493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1645.168291] LNet: 91220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1645.203477] LNetError: 91220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1645.212692] LNetError: 91220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1645.234068] LNet: Removed LNI 192.168.202.35@tcp [ 1645.288808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1645.415762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1645.478301] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1654.499088] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:43:18 (1763343798) [ 1654.658297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.686462] LNet: 91930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.700848] LNetError: 91930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.714976] LNet: Removed LNI 192.168.202.35@tcp [ 1654.735445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.786990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.823724] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1663.553209] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:43:27 (1763343807) [ 1665.249150] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1667.143673] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:43:31 (1763343811) [ 1667.908782] LNet: 92965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.911778] LNetError: 92965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1667.936520] LNet: Removed LNI 192.168.202.35@tcp [ 1668.606458] Key type .llcrypt unregistered [ 1668.607861] Key type ._llcrypt unregistered [ 1674.636972] Key type ._llcrypt registered [ 1674.645743] Key type .llcrypt registered [ 1674.922590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.942163] alg: No test for adler32 (adler32-zlib) [ 1675.918888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1675.959866] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1683.430772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1688.268093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.135@tcp [ 1688.687409] LNet: 93378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.690365] LNetError: 93378:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.712013] LNet: Removed LNI 192.168.202.35@tcp [ 1689.180751] Key type .llcrypt unregistered [ 1689.183368] Key type ._llcrypt unregistered [ 1700.462997] Key type ._llcrypt registered [ 1700.465718] Key type .llcrypt registered [ 1702.086442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1703.897917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1705.725389] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:44:10 (1763343850) [ 1706.331615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.349447] alg: No test for adler32 (adler32-zlib) [ 1707.319183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.468274] LNet: 94462:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.229492] Key type .llcrypt unregistered [ 1708.234230] Key type ._llcrypt unregistered [ 1714.015213] Key type ._llcrypt registered [ 1714.017906] Key type .llcrypt registered [ 1714.339910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1714.363550] alg: No test for adler32 (adler32-zlib) [ 1715.379963] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1715.382806] LNet: 94714:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1715.426171] LNetError: 94726:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 1715.440093] LNetError: 94726:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.36@tcp has gone from up to down [ 1715.954703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.984530] LNet: 94772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1715.996108] LNetError: 94772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.023853] LNet: Removed LNI 192.168.202.35@tcp [ 1716.079936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.179910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1716.205435] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1716.211454] LNet: 94779:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1716.216509] LNet: 94779:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1716.220854] LNetError: 94779:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.36@tcp is a gateway. Can not delete it [ 1716.240335] LNetError: 94779:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.202.36@tcp is a router. Can not be deleted [ 1716.257808] LNet: 94779:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.260122] LNetError: 94776:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.40@tcp is being used as a gateway but routing feature is not turned on [ 1716.354549] LNetError: 94776:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1716.364494] LNetError: 94776:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.40@tcp has gone from up to down [ 1716.374130] LNetError: 94776:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1724.038800] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:44:28 (1763343868) [ 1726.207653] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg235-server [ 1728.314588] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:44:32 (1763343872) [ 1730.434773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg235-server [ 1732.791213] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:44:36 (1763343876) [ 1734.446724] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 1736.405113] Key type lgssc registered [ 1737.824551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1756.961317] Key type lgssc unregistered [ 1757.283383] LNet: 97197:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.292134] LNetError: 97197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.319106] LNet: Removed LNI 192.168.202.35@tcp [ 1757.794235] Key type .llcrypt unregistered [ 1757.795719] Key type ._llcrypt unregistered [ 1771.389270] Key type ._llcrypt registered [ 1771.390978] Key type .llcrypt registered [ 1772.791112] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:45:17 (1763343917) [ 1773.195131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.222980] alg: No test for adler32 (adler32-zlib) [ 1774.163574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.242713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1774.306362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1774.350466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1774.403491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1780.862196] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:45:25 (1763343925) [ 1781.009812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.071412] LNet: 98611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.113135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.170750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.196504] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1782.018913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.043692] LNet: 98759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.052299] LNetError: 98759:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.069519] LNet: Removed LNI 192.168.202.35@tcp [ 1782.101159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.153813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1782.182860] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1782.628789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.651220] LNetError: 98787:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.666257] LNet: Removed LNI 192.168.202.35@tcp [ 1782.700625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.775532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1782.802610] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1789.704464] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:45:34 (1763343934) [ 1790.055158] LNet: 99377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.063384] LNet: 99377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1790.075505] LNetError: 99377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.094638] LNet: Removed LNI 192.168.202.35@tcp [ 1790.895037] Key type .llcrypt unregistered [ 1790.913598] Key type ._llcrypt unregistered [ 1797.182362] Key type ._llcrypt registered [ 1797.183988] Key type .llcrypt registered [ 1797.400611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1797.424375] alg: No test for adler32 (adler32-zlib) [ 1798.443547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.587367] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1798.595685] LNet: *** cfs_fail_loc=e003, val=100*** [ 1799.540604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1799.574039] LNet: *** cfs_fail_loc=e003, val=100*** [ 1799.584853] LNet: Skipped 2 previous similar messages [ 1799.596111] Ping reply received with mapping: [ 1799.606291] found metadata reply NI[192.168.202.35@tcp] [ 1799.611035] 23 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1799.615069] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1799.622524] 5a 5a 5a 5a 5a 5a 5a 5a [ 1799.793293] LNet: 99772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.797924] LNetError: 99772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.876990] LNet: Removed LNI 192.168.202.35@tcp [ 1801.847149] Key type .llcrypt unregistered [ 1801.848942] Key type ._llcrypt unregistered [ 1814.124461] Key type ._llcrypt registered [ 1814.126054] Key type .llcrypt registered [ 1815.601290] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:45:59 (1763343959) [ 1816.144447] Key type .llcrypt unregistered [ 1816.146250] Key type ._llcrypt unregistered [ 1821.959539] Key type ._llcrypt registered [ 1821.962145] Key type .llcrypt registered [ 1822.182627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1822.212641] alg: No test for adler32 (adler32-zlib) [ 1823.445188] LNetError: 100807:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1823.818822] Key type .llcrypt unregistered [ 1823.820966] Key type ._llcrypt unregistered [ 1829.686780] Key type ._llcrypt registered [ 1829.688630] Key type .llcrypt registered [ 1830.912383] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:46:15 (1763343975) [ 1831.607516] Key type .llcrypt unregistered [ 1831.609322] Key type ._llcrypt unregistered [ 1836.303701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1836.310633] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1838.965184] Key type ._llcrypt registered [ 1838.966638] Key type .llcrypt registered [ 1839.177225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1839.198105] alg: No test for adler32 (adler32-zlib) [ 1840.171106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.231828] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1849.869893] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:46:34 (1763343994) [ 1850.004433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.042605] LNet: 102576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.049275] LNetError: 102576:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1850.062907] LNet: Removed LNI 10.1.2.3@tcp [ 1850.102367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.808813] Key type .llcrypt unregistered [ 1850.810806] Key type ._llcrypt unregistered [ 1855.903103] Key type ._llcrypt registered [ 1855.905216] Key type .llcrypt registered [ 1856.114339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1856.135360] alg: No test for adler32 (adler32-zlib) [ 1857.067662] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1857.119482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1857.319073] LNet: 102864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.330019] LNetError: 102864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.349715] LNet: Removed LNI 192.168.202.35@tcp [ 1859.045640] Key type .llcrypt unregistered [ 1859.047166] Key type ._llcrypt unregistered [ 1869.788236] Key type ._llcrypt registered [ 1869.796927] Key type .llcrypt registered [ 1871.305722] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:46:55 (1763344015) [ 1872.108533] Key type .llcrypt unregistered [ 1872.112190] Key type ._llcrypt unregistered [ 1877.739117] Key type ._llcrypt registered [ 1877.740717] Key type .llcrypt registered [ 1878.024417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1878.045880] alg: No test for adler32 (adler32-zlib) [ 1884.570940] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:47:09 (1763344029) [ 1884.890343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.947760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.013334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.057380] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1885.903415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.943059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1885.973805] LNet: Removed LNI 192.168.202.35@tcp [ 1886.194123] LNet: 104600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.554065] Key type .llcrypt unregistered [ 1886.559153] Key type ._llcrypt unregistered [ 1895.724614] Key type ._llcrypt registered [ 1895.725984] Key type .llcrypt registered [ 1897.077917] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:47:21 (1763344041) [ 1897.476743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.507530] alg: No test for adler32 (adler32-zlib) [ 1898.433972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.485807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.503577] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1899.116798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1899.145939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.35@tcp [ 1899.162429] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1899.164361] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1899.166701] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1899, 0, 1949 [ 1899.169642] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 1899.171160] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1899.173405] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1899.174922] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1899, 0, 1949 [ 1899.176750] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 1904.689404] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:47:29 (1763344049) [ 1905.087520] LNet: 106136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.091457] LNetError: 106136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.102346] LNet: Removed LNI 192.168.202.35@tcp [ 1905.606680] Key type .llcrypt unregistered [ 1905.608717] Key type ._llcrypt unregistered [ 1909.977280] Key type ._llcrypt registered [ 1909.978544] Key type .llcrypt registered [ 1910.130518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1910.142278] alg: No test for adler32 (adler32-zlib) [ 1911.046432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1911.078682] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1918.483598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1926.042632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1926.104494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1926.149493] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1927.381844] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1927.580132] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1927.601803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1927.610381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1929.972320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1930.002763] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1941.158367] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:48:05 (1763344085) [ 1941.248584] LNet: 107629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.278797] LNetError: 107629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.304261] LNet: Removed LNI 192.168.202.35@tcp [ 1942.305990] LNetError: 107629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1942.314255] LNet: Removed LNI 192.168.202.35@tcp1 [ 1942.324787] LNet: Removed LNI 10.1.2.3@tcp99 [ 1943.092473] Key type .llcrypt unregistered [ 1943.094232] Key type ._llcrypt unregistered [ 1948.195502] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1948.209551] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1950.779658] Key type ._llcrypt registered [ 1950.782313] Key type .llcrypt registered [ 1950.954466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1950.969331] alg: No test for adler32 (adler32-zlib) [ 1951.898351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.996331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1952.028886] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1952.918700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1952.944990] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1955.280940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1955.326970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1955.387542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1955.406528] LNetError: 108378:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1955.439552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1955.486642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1956.006560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1956.110402] LNet: 108398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.118921] LNetError: 108398:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.131087] LNet: Removed LNI 192.168.202.35@tcp [ 1956.145967] LNet: Removed LNI 10.1.2.3@tcp [ 1956.892955] Key type .llcrypt unregistered [ 1956.897637] Key type ._llcrypt unregistered [ 1967.892414] Key type ._llcrypt registered [ 1967.893845] Key type .llcrypt registered [ 1969.185837] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:48:33 (1763344113) [ 1969.432766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1969.440571] alg: No test for adler32 (adler32-zlib) [ 1970.349850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1970.404939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1970.452666] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1971.140837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1971.196472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg235-client@tcp [ 1977.682942] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:48:42 (1763344122) [ 1977.756675] LNet: 109918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.760520] LNetError: 109918:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1978.790536] LNet: Removed LNI 192.168.202.35@tcp [ 1979.402655] Key type .llcrypt unregistered [ 1979.404496] Key type ._llcrypt unregistered [ 1984.218612] Key type ._llcrypt registered [ 1984.220051] Key type .llcrypt registered [ 1984.425733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.453212] alg: No test for adler32 (adler32-zlib) [ 1985.413754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1985.450797] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1985.501424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1992.806980] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:48:57 (1763344137) [ 1994.578940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1996.449860] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:49:01 (1763344141) [ 1998.318369] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2000.159613] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:49:04 (1763344144) [ 2000.256378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2000.286564] LNet: 111335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.293296] LNetError: 111335:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2000.303663] LNet: Removed LNI 192.168.202.35@tcp [ 2000.341250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.403326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2000.425668] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2001.197720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2001.264224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2008.294113] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:49:13 (1763344153) [ 2008.537507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.567309] LNet: 112051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.575751] LNetError: 112051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.593770] LNet: Removed LNI 192.168.202.35@tcp [ 2008.610214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2008.667535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2008.698838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2009.395141] Key type .llcrypt unregistered [ 2009.396587] Key type ._llcrypt unregistered [ 2020.557233] Key type ._llcrypt registered [ 2020.569102] Key type .llcrypt registered [ 2022.303827] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:49:26 (1763344166) [ 2022.854295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2022.877986] alg: No test for adler32 (adler32-zlib) [ 2023.890293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2023.967844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2024.035036] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2024.929502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2024.972434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2031.227428] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:49:35 (1763344175) [ 2031.473339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.492299] LNet: 113603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.502583] LNetError: 113603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.520377] LNet: Removed LNI 192.168.202.35@tcp [ 2031.545731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.595887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2031.625669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2038.425430] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:49:42 (1763344182) [ 2038.504560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.518909] LNet: 114178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.573554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.698896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2039.190927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2039.226763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2039.271505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2039.664535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2039.720202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2039.847571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2039.883908] LNet: 114218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.922543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.036107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2040.364422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2040.414732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2040.510679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2040.844478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2040.887430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2047.312381] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:49:51 (1763344191) [ 2047.426039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2047.458422] LNet: 114810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.514140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2047.625319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2047.660431] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2048.524921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2048.552406] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2049.486748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2049.601741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2049.897076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2049.926888] LNetError: 115098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.976196] LNet: Removed LNI 192.168.202.35@tcp [ 2050.981826] LNetError: 115098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.003392] LNet: Removed LNI 192.168.202.35@tcp2 [ 2051.036733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.132663] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2051.144612] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2051.151478] LNet: 115104:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2051.907819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2052.154560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2052.179434] LNet: 115141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2052.186750] LNet: 115141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2053.217871] LNet: Removed LNI 192.168.202.35@tcp [ 2053.227402] LNetError: 115141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2053.243991] LNetError: 115141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2053.270774] LNet: Removed LNI 192.168.202.35@tcp2 [ 2053.306412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2053.420678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2053.490324] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2054.235436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2054.250955] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2055.122147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2055.249392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2055.333904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2055.667206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2056.742182] LNet: Removed LNI 192.168.202.35@tcp [ 2057.770832] LNet: Removed LNI 192.168.202.35@tcp2 [ 2057.795380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2057.875095] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2057.882755] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2057.890624] LNet: 115432:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2058.586231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2058.822824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.858742] LNetError: 115469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.863154] LNetError: 115469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2058.873211] LNet: Removed LNI 192.168.202.35@tcp [ 2059.881667] LNet: Removed LNI 192.168.202.35@tcp2 [ 2059.911214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.027195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2060.085677] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2061.051330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2061.085240] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2062.310936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2062.383736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2062.665505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2062.694342] LNet: 115749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2062.698544] LNet: 115749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2063.712469] LNet: Removed LNI 192.168.202.35@tcp [ 2063.733181] LNet: Removed LNI 192.168.202.35@tcp2 [ 2063.787436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2063.950960] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2063.956086] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2063.962763] LNet: 115755:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2064.691604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2065.039653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2066.153312] LNet: Removed LNI 192.168.202.35@tcp [ 2066.174629] LNet: Removed LNI 192.168.202.35@tcp2 [ 2066.227939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.365389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2066.417518] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2067.642212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2067.659811] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2068.999282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2069.061691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2069.160616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2069.520415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.553857] LNetError: 116078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.557234] LNetError: 116078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2070.560300] LNet: Removed LNI 192.168.202.35@tcp [ 2071.590972] LNet: Removed LNI 192.168.202.35@tcp2 [ 2071.605879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.735455] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2071.742404] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2071.751217] LNet: 116084:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2072.758241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2073.113673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.156286] LNet: Removed LNI 192.168.202.35@tcp [ 2074.218855] LNet: Removed LNI 192.168.202.35@tcp2 [ 2074.262036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.367416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.427326] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2075.761345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2075.798563] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2076.705564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2076.725721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2076.774221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2077.082813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.113173] LNet: Removed LNI 192.168.202.35@tcp [ 2078.126264] LNet: Removed LNI 192.168.202.35@tcp2 [ 2078.169954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.248284] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2078.257473] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2078.270258] LNet: 116424:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2078.824915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2079.177518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.197189] LNet: 116476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.201573] LNet: 116476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2080.224800] LNet: Removed LNI 192.168.202.35@tcp [ 2080.235919] LNet: Removed LNI 192.168.202.35@tcp2 [ 2080.267243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.364111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.426817] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2081.193186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2081.215632] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2082.200336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2082.261339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2082.289505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2082.360903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2082.732447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.815897] LNet: Removed LNI 192.168.202.35@tcp [ 2084.845138] LNet: Removed LNI 192.168.202.35@tcp2 [ 2084.878864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.974111] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2084.984258] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 2085.618426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2092.325358] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2094.176736] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:50:38 (1763344238) [ 2094.249371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.286768] LNetError: 117538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.298097] LNetError: 117538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2094.306755] LNet: Removed LNI 192.168.202.35@tcp [ 2095.342277] LNet: Removed LNI 192.168.202.35@tcp2 [ 2095.376102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.659129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2095.685725] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2096.392675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2096.406424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2097.422191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.457389] LNet: Removed LNI 192.168.202.35@tcp [ 2097.478386] LNet: Removed LNI 10.1.2.3@tcp [ 2097.502771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.572754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2097.596969] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2097.601625] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2098.004989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2098.039772] LNet: Removed LNI 192.168.202.35@tcp [ 2098.054568] LNet: Removed LNI 10.1.2.3@tcp [ 2105.007796] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:50:49 (1763344249) [ 2105.087707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2105.143804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.318476] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2105.478913] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2105.487696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2105.500478] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2107.967875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2108.017159] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2108.813643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2108.832091] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2110.924183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 2110.966809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2111.027095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.35@tcp -m PUT -e local_timeout -r 1 [ 2111.090625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2111.200969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2111.206956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2111.457137] LNetError: 118487:0:(lib-move.c:4947:lnet_parse()) 192.168.202.35@tcp, src 192.168.202.35@tcp: Dropping PUT (error -108 looking up sender) [ 2112.485406] LNet: 118945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.488656] LNet: 118945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2113.504240] LNet: Removed LNI 192.168.202.35@tcp [ 2113.518958] LNet: Removed LNI 10.1.2.3@tcp [ 2113.961349] Key type .llcrypt unregistered [ 2113.966483] Key type ._llcrypt unregistered [ 2124.677970] Key type ._llcrypt registered [ 2124.679828] Key type .llcrypt registered [ 2125.861637] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:51:10 (1763344270) [ 2126.365078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.392116] alg: No test for adler32 (adler32-zlib) [ 2127.338633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.728042] LNet: 119795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.374902] Key type .llcrypt unregistered [ 2128.377602] Key type ._llcrypt unregistered [ 2134.031094] Key type ._llcrypt registered [ 2134.038040] Key type .llcrypt registered [ 2134.322649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2134.335029] alg: No test for adler32 (adler32-zlib) [ 2135.287910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2135.322858] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2141.302604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2141.508091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2142.170889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2142.472517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2142.503419] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2142.518109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2142.567929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2142.606639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2142.656075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2142.715173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2142.767951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2142.805627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2142.842481] LNet: 120240:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2142.864995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2142.911996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2142.956730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2143.027498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2143.069512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2143.125877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2143.835802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2144.113613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2144.155496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2144.222601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2144.270305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2144.320390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2144.371625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2144.415278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2144.454132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2144.470602] LNet: 120357:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2144.499187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2144.555977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2144.604975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2144.641297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2144.709577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2144.758124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2145.421256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2145.674081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2145.716928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2145.743902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2145.800564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2145.862561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2145.903070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2145.957911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2146.006130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2146.040339] LNet: 120475:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2146.077748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2146.129281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2146.171713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2146.199525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2146.233529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2146.290771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2146.997930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2147.252511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2147.284416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2147.324512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2147.357587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2147.388926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2147.450684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2147.500307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2147.567369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2147.601979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2147.649188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2147.683787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2147.715388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2147.802352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2147.838661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2148.209829] LNet: 120637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2148.217140] LNetError: 120637:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2148.229635] LNet: Removed LNI 192.168.202.35@tcp [ 2148.676257] Key type .llcrypt unregistered [ 2148.677926] Key type ._llcrypt unregistered [ 2165.698359] Key type ._llcrypt registered [ 2165.700218] Key type .llcrypt registered [ 2168.010659] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:51:51 (1763344311) [ 2168.482060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.502655] alg: No test for adler32 (adler32-zlib) [ 2169.455304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.089774] LNet: 121521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2171.135592] Key type .llcrypt unregistered [ 2171.138447] Key type ._llcrypt unregistered [ 2178.513449] Key type ._llcrypt registered [ 2178.514990] Key type .llcrypt registered [ 2178.803394] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2178.818201] alg: No test for adler32 (adler32-zlib) [ 2179.795303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2179.825567] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2186.421361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2193.167458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.249174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2193.289515] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2194.295133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2194.991136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2195.228977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2195.234308] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2195.260794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2195.299550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2195.366712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2195.398403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2195.454887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2195.503818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2195.569334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2195.599738] LNet: 122169:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2195.639384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2195.702410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2195.790605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2195.853618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2195.881284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2195.916057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2196.642258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2197.004788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2197.054186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2197.089278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2197.128825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2197.169659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2197.202869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2197.239583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2197.263600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2197.285824] LNet: 122287:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2197.311796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2197.341232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2197.372471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2197.410719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2197.445422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2197.479267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2198.254470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2198.543086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2198.638587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2198.688357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2198.720903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2198.764374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2198.804623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2198.847725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2198.896023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2198.929420] LNet: 122404:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.973910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2199.005107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2199.044830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2199.080242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2199.105792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2199.139472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2200.014781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2200.319480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2200.345704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2200.379030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2200.416150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2200.435612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2200.465123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2200.490854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2200.516568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2200.542715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2200.567425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2200.591205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2200.654409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2200.714737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2200.816830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2204.858504] LNet: 122601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.869122] LNetError: 122601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.922142] LNet: Removed LNI 192.168.202.35@tcp [ 2205.931769] LNetError: 122601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2205.950617] LNet: Removed LNI 192.168.202.35@tcp1 [ 2206.453890] Key type .llcrypt unregistered [ 2206.456248] Key type ._llcrypt unregistered [ 2221.401394] Key type ._llcrypt registered [ 2221.403486] Key type .llcrypt registered [ 2223.045392] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1996 sec ========== 20:52:47 (1763344367) [ 2223.701824] Key type .llcrypt unregistered [ 2223.706891] Key type ._llcrypt unregistered [ 2234.196428] Key type ._llcrypt registered [ 2234.198048] Key type .llcrypt registered [ 2234.577535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2234.601176] alg: No test for adler32 (adler32-zlib) [ 2235.569954] Lustre: Lustre: Build Version: 2.16.61_47_g0e58e73 [ 2235.748420] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2237.424462] Key type lgssc registered [ 2238.251047] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2288.242557] Lustre: Mounted lustre-client [ 2292.749735] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2304.131935] LustreError: 125825:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff90f2ca9ce000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2304.146124] LustreError: 125825:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2304.184177] Lustre: Unmounted lustre-client [ 2321.975497] Key type lgssc unregistered [ 2322.252866] LNet: 126303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2322.259742] LNetError: 126303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2322.275706] LNet: Removed LNI 192.168.202.35@tcp [ 2322.857289] Key type .llcrypt unregistered [ 2322.866761] Key type ._llcrypt unregistered