[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 464946547 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003084] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007629] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010135] LSM: Security Framework initializing [ 0.012037] Yama: becoming mindful. [ 0.013102] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.022818] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027639] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028189] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031150] *** VALIDATE tmpfs *** [ 0.033172] *** VALIDATE proc *** [ 0.034295] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036313] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037180] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039046] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043174] debug: unmapping init [mem 0xffffffff98259000-0xffffffff98260fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046767] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047031] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050025] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054401] rcu: Hierarchical SRCU implementation. [ 0.056680] smp: Bringing up secondary CPUs ... [ 0.057770] x86: Booting SMP configuration: [ 0.058045] .... node #0, CPUs: #1 #2 #3 [ 0.062071] smp: Brought up 1 node, 4 CPUs [ 0.064020] smpboot: Max logical packages: 1 [ 0.065023] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.259108] node 0 deferred pages initialised in 192ms [ 0.263111] devtmpfs: initialized [ 0.264333] x86/mm: Memory block size: 128MB [ 0.267887] gcov: version magic: 0x41383552 [ 0.270406] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.273125] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.275460] pinctrl core: initialized pinctrl subsystem [ 0.278358] [ 0.278922] ************************************************************* [ 0.281015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.283013] ** ** [ 0.285012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.287018] ** ** [ 0.289014] ** This means that this kernel is built to expose internal ** [ 0.291017] ** IOMMU data structures, which may compromise security on ** [ 0.293016] ** your system. ** [ 0.295016] ** ** [ 0.297016] ** If you see this message and you are not debugging the ** [ 0.300017] ** kernel, report this immediately to your vendor! ** [ 0.302027] ** ** [ 0.304018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.305013] ************************************************************* [ 0.307518] NET: Registered protocol family 16 [ 0.309559] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.312093] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.316099] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320057] cpuidle: using governor menu [ 0.321869] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.324581] PCI: Using configuration type 1 for base access [ 0.326166] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.336183] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.339048] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.343263] cryptd: max_cpu_qlen set to 1000 [ 0.346292] ACPI: Added _OSI(Module Device) [ 0.348017] ACPI: Added _OSI(Processor Device) [ 0.349015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.351017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.354318] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.360595] ACPI: Interpreter enabled [ 0.362072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.363018] ACPI: Using IOAPIC for interrupt routing [ 0.365175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.368464] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.378419] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.381056] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.383024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.386096] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.391181] acpiphp: Slot [2] registered [ 0.393114] acpiphp: Slot [3] registered [ 0.394091] acpiphp: Slot [4] registered [ 0.395161] acpiphp: Slot [5] registered [ 0.397173] acpiphp: Slot [6] registered [ 0.398156] acpiphp: Slot [7] registered [ 0.400120] acpiphp: Slot [8] registered [ 0.402172] acpiphp: Slot [9] registered [ 0.403136] acpiphp: Slot [10] registered [ 0.405163] acpiphp: Slot [11] registered [ 0.407132] acpiphp: Slot [12] registered [ 0.408144] acpiphp: Slot [13] registered [ 0.410109] acpiphp: Slot [14] registered [ 0.411128] acpiphp: Slot [15] registered [ 0.413121] acpiphp: Slot [16] registered [ 0.414112] acpiphp: Slot [17] registered [ 0.416153] acpiphp: Slot [18] registered [ 0.417114] acpiphp: Slot [19] registered [ 0.418129] acpiphp: Slot [20] registered [ 0.420127] acpiphp: Slot [21] registered [ 0.421147] acpiphp: Slot [22] registered [ 0.423126] acpiphp: Slot [23] registered [ 0.424143] acpiphp: Slot [24] registered [ 0.426104] acpiphp: Slot [25] registered [ 0.427138] acpiphp: Slot [26] registered [ 0.428017] acpiphp: Slot [27] registered [ 0.429149] acpiphp: Slot [28] registered [ 0.431132] acpiphp: Slot [29] registered [ 0.432121] acpiphp: Slot [30] registered [ 0.434131] acpiphp: Slot [31] registered [ 0.435081] PCI host bridge to bus 0000:00 [ 0.437028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.439038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.441034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.444037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.447037] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.449038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.451244] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.455000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.460551] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.467913] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.470909] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.473023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.475020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.477021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.479518] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.481800] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.484054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.487795] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.491682] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.498984] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.503978] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.509363] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.516019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.523029] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.543027] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.551454] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.558037] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.573024] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.583842] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.586453] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.589373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.590300] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.592296] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.598147] iommu: Default domain type: Passthrough [ 0.599378] SCSI subsystem initialized [ 0.600089] ACPI: bus type USB registered [ 0.601090] usbcore: registered new interface driver usbfs [ 0.603054] usbcore: registered new interface driver hub [ 0.604059] usbcore: registered new device driver usb [ 0.605149] pps_core: LinuxPPS API ver. 1 registered [ 0.606008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.608055] PTP clock support registered [ 0.610167] EDAC MC: Ver: 3.0.0 [ 0.612197] PCI: Using ACPI for IRQ routing [ 0.613843] NetLabel: Initializing [ 0.615016] NetLabel: domain hash size = 128 [ 0.617014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.619106] NetLabel: unlabeled traffic allowed by default [ 0.622138] vgaarb: loaded [ 0.624085] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.625012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.632000] clocksource: Switched to clocksource kvm-clock [ 0.741464] VFS: Disk quotas dquot_6.6.0 [ 0.743103] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.745368] *** VALIDATE ramfs *** [ 0.746502] *** VALIDATE hugetlbfs *** [ 0.747777] pnp: PnP ACPI init [ 0.750263] pnp: PnP ACPI: found 6 devices [ 0.764887] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.767838] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.769983] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.771742] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.773755] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.775892] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.778539] NET: Registered protocol family 2 [ 0.780349] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.785246] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.788606] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.793769] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.797134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.800189] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.803377] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806235] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.809260] NET: Registered protocol family 1 [ 0.812097] RPC: Registered named UNIX socket transport module. [ 0.814240] RPC: Registered udp transport module. [ 0.815712] RPC: Registered tcp transport module. [ 0.817480] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.819904] NET: Registered protocol family 44 [ 0.821303] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.822791] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.824597] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.826604] PCI: CLS 0 bytes, default 64 [ 0.827980] Unpacking initramfs... [ 2.254698] debug: unmapping init [mem 0xffff8ce07cc64000-0xffff8ce07ffcffff] [ 2.258541] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.260771] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.263607] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.796618] Initialise system trusted keyrings [ 2.798540] Key type blacklist registered [ 2.802182] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.813184] zbud: loaded [ 2.817228] *** VALIDATE nfs *** [ 2.818809] *** VALIDATE nfs4 *** [ 2.820388] pstore: using deflate compression [ 2.826078] Platform Keyring initialized [ 2.955950] NET: Registered protocol family 38 [ 2.957722] Key type asymmetric registered [ 2.959115] Asymmetric key parser 'x509' registered [ 2.961115] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.964574] io scheduler mq-deadline registered [ 2.967124] io scheduler kyber registered [ 2.968738] io scheduler bfq registered [ 2.970688] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.973288] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.975789] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.978228] ACPI: Power Button [PWRF] [ 3.079290] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.174854] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.270644] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.301939] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.333831] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.339365] Non-volatile memory driver v1.3 [ 3.340780] Linux agpgart interface v0.103 [ 3.375103] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.377655] vda: detected capacity change from 0 to 68530176 [ 3.391408] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.394484] vdb: detected capacity change from 0 to 1073741824 [ 3.400062] libphy: Fixed MDIO Bus: probed [ 3.407812] usbcore: registered new interface driver usbserial_generic [ 3.410410] usbserial: USB Serial support registered for generic [ 3.413629] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.418698] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.421551] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.423812] mousedev: PS/2 mouse device common for all mice [ 3.427670] rtc_cmos 00:05: RTC can wake from S4 [ 3.436274] rtc_cmos 00:05: registered as rtc0 [ 3.437975] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.441094] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.443524] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.446688] intel_pstate: CPU model not supported [ 3.453506] hid: raw HID events driver (C) Jiri Kosina [ 3.454047] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.455505] usbcore: registered new interface driver usbhid [ 3.459944] usbhid: USB HID core driver [ 3.461208] drop_monitor: Initializing network drop monitor service [ 3.463487] Initializing XFRM netlink socket [ 3.465796] NET: Registered protocol family 10 [ 3.468836] Segment Routing with IPv6 [ 3.469910] NET: Registered protocol family 17 [ 3.472211] mpls_gso: MPLS GSO support [ 3.477323] RAS: Correctable Errors collector initialized. [ 3.479264] AVX version of gcm_enc/dec engaged. [ 3.480578] AES CTR mode by8 optimization enabled [ 3.566149] sched_clock: Marking stable (3566104850, 0)->(4492073544, -925968694) [ 3.569279] registered taskstats version 1 [ 3.571294] Loading compiled-in X.509 certificates [ 3.573398] zswap: loaded using pool lzo/zbud [ 3.598171] Key type big_key registered [ 3.612378] Key type encrypted registered [ 3.614018] ima: No TPM chip found, activating TPM-bypass! [ 3.616432] ima: Allocated hash algorithm: sha1 [ 3.618028] ima: No architecture policies found [ 3.619364] evm: Initialising EVM extended attributes: [ 3.621527] evm: security.selinux [ 3.622646] evm: security.ima [ 3.623773] evm: security.capability [ 3.624961] evm: HMAC attrs: 0x1 [ 3.627265] rtc_cmos 00:05: setting system clock to 2025-11-17 01:54:44 UTC (1763344484) [ 3.634285] debug: unmapping init [mem 0xffffffff99203000-0xffffffff993fffff] [ 3.637046] debug: unmapping init [mem 0xffffffff97f82000-0xffffffff98258fff] [ 3.645094] Write protecting the kernel read-only data: 28672k [ 3.650280] debug: unmapping init [mem 0xffffffff96603000-0xffffffff967fffff] [ 3.652434] debug: unmapping init [mem 0xffffffff96f14000-0xffffffff96ffffff] [ 3.689726] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.698573] systemd[1]: Detected virtualization kvm. [ 3.700390] systemd[1]: Detected architecture x86-64. [ 3.702215] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.734159] systemd[1]: No hostname configured. [ 3.735467] systemd[1]: Set hostname to . [ 3.736686] random: systemd: uninitialized urandom read (16 bytes read) [ 3.737863] systemd[1]: Initializing machine ID from random generator. [ 3.780375] random: ln: uninitialized urandom read (6 bytes read) [ 3.868657] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871698] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.881258] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.887392] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.527194] device-mapper: uevent: version 1.0.3 [ 4.529438] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.351610] virtio_net virtio0 ens2: renamed from eth0 [ 5.385345] scsi host0: ata_piix [ 5.403120] scsi host1: ata_piix [ 5.406242] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.408331] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.753981] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.130228] random: crng init done [ 10.131408] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.651425] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.930890] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.192248] SELinux: Disabled at runtime. [ 12.245968] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.254443] systemd[1]: Detected virtualization kvm. [ 12.255995] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.797776] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.802141] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.806559] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.811156] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.815224] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.827542] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.833913] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 12.938508] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.366526] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.699290] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.727446] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.833862] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.857781] EDAC sbridge: Ver: 1.1.2 [ 15.374797] Key type dns_resolver registered [ 15.696175] NFS: Registering the id_resolver key type [ 15.698054] Key type id_resolver registered [ 15.699337] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg235-client login: [ 47.662345] libcfs: loading out-of-tree module taints kernel. [ 47.679691] Key type ._llcrypt registered [ 47.692052] Key type .llcrypt registered [ 47.987981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.000895] alg: No test for adler32 (adler32-zlib) [ 49.160749] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 49.705341] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 51.391183] Key type lgssc registered [ 52.688545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 136.760006] hrtimer: interrupt took 6539624 ns [ 152.851594] Lustre: Mounted lustre-client [ 156.506704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 177.181322] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing check_logdir /tmp/testlogs/ [ 178.660618] Lustre: lustre-OST0000-osc-ffff8ce0c9010000: disconnect after 24s idle [ 180.502475] Lustre: DEBUG MARKER: oleg235-client.virtnet: executing yml_node [ 183.856525] Lustre: DEBUG MARKER: Client: 2.16.61.46 [ 185.877422] Lustre: DEBUG MARKER: MDS: 2.16.61.46 [ 187.905205] Lustre: DEBUG MARKER: OSS: 2.16.61.46 [ 189.189146] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 20:57:48 EST 2025 [ 202.921284] LustreError: 7938:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ce0c9010000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 202.946989] LustreError: 7938:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 202.996332] Lustre: Unmounted lustre-client [ 238.614492] Key type lgssc unregistered [ 238.899490] LNet: 8386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 238.909548] LNetError: 8386:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 238.938320] LNet: Removed LNI 192.168.202.35@tcp [ 239.737220] Key type .llcrypt unregistered [ 239.747864] Key type ._llcrypt unregistered [ 259.319381] Key type ._llcrypt registered [ 259.322914] Key type .llcrypt registered [ 259.504810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 259.528459] alg: No test for adler32 (adler32-zlib) [ 260.428574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 260.485455] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 260.500713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 264.796709] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 264.964429] LNet: 9252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 264.981291] LNetError: 9252:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 265.032860] LNet: Removed LNI 192.168.202.35@tcp [ 265.703461] Key type .llcrypt unregistered [ 265.715816] Key type ._llcrypt unregistered [ 271.075488] Key type ._llcrypt registered [ 271.084385] Key type .llcrypt registered [ 272.562440] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:59:11 (1763344751) [ 272.910204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 272.936397] alg: No test for adler32 (adler32-zlib) [ 273.806980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.864880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.894723] LNet: 9665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 273.928554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.996551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 274.011356] LNet: 9673:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 279.910996] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:59:19 (1763344759) [ 280.169771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.195679] LNet: 10249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.219903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.304609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 285.521772] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:59:25 (1763344765) [ 285.642213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.666434] LNet: 10828:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 285.715224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.778483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 290.651581] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:59:30 (1763344770) [ 290.725344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.737475] LNet: 11414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.751303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.817834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 295.836432] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:59:35 (1763344775) [ 295.900176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.927149] LNet: 11993:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.956452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.047663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 296.447175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 301.760934] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:59:41 (1763344781) [ 301.917473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.939380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.992404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 302.358817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 302.724632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 307.691698] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:59:47 (1763344787) [ 307.766408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.786276] LNet: 13187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.793539] LNet: 13187:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 307.817868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.889733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 313.351262] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:59:53 (1763344793) [ 313.549467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.574871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.647695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 313.674296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 314.091663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 314.128437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 314.648795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 314.670465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 314.987402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 315.024384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 315.457969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 315.474854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 316.071281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 316.124501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 316.730581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 316.761798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 317.199501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 317.241352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 317.699950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 317.745898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 318.142799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 318.203865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 318.658645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 318.687576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 318.956919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 318.995163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 319.310155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 319.338829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 325.964139] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 21:00:05 (1763344805) [ 326.009433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.035856] LNet: 14446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.042126] LNet: 14446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 326.109559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.240852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 326.272332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 332.293991] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 21:00:11 (1763344811) [ 332.569200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.612902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.702795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 332.728368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 338.233886] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 21:00:18 (1763344818) [ 338.393443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.463963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.536349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 338.565690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 343.575084] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 21:00:23 (1763344823) [ 343.660150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.725530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.808978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 343.839973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 350.042777] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 21:00:29 (1763344829) [ 350.148178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.264158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.315228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 350.341360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 356.315200] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 21:00:36 (1763344836) [ 356.446228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.506960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.590552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 356.612375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 363.058305] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 21:00:42 (1763344842) [ 363.204949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.239825] LNet: 17972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.241884] LNet: 17972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 363.269517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.366357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.783603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 369.854516] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 21:00:49 (1763344849) [ 370.014318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.092137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.298551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 370.550847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 377.152844] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 21:00:56 (1763344856) [ 377.245715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.302941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.518165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 377.764648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 384.600870] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 21:01:04 (1763344864) [ 384.801022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.879698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.293600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 385.570757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 392.495561] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 21:01:11 (1763344871) [ 392.806126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.864231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.948835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 393.028350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 399.852479] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 21:01:19 (1763344879) [ 400.210480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.275340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.344741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.387992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 407.165922] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 21:01:26 (1763344886) [ 407.249303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.308647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.408905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 413.482533] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 21:01:33 (1763344893) [ 413.736382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.784508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.857541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 420.903122] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 21:01:40 (1763344900) [ 421.044319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 421.134500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.248415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 421.554401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 427.694655] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 21:01:47 (1763344907) [ 427.827267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.922887] LNet: 23427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.925332] LNet: 23427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 427.948893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.004523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 428.079073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 435.016866] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 21:01:54 (1763344914) [ 435.120900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.175951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.275201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 435.305841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 442.305100] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 21:02:01 (1763344921) [ 442.400495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.484569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.562245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 442.595415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 448.981418] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 21:02:08 (1763344928) [ 449.195881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.242893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.312918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 449.356248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 455.685176] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 21:02:15 (1763344935) [ 455.858343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.899601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.946126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 455.994413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 456.029289] LNetError: 25787:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 456.074816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 456.106754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 462.408659] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 21:02:21 (1763344941) [ 462.639140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.715958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.800505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ddwdgb2vrL [ 462.870863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ddwdgb2vrL [ 462.952156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ddwdgb2vrL [ 469.184827] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 21:02:28 (1763344948) [ 469.507955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.564749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.610810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 469.656680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 469.715682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 475.582715] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 21:02:35 (1763344955) [ 475.714544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 475.755318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 475.893922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 478.467902] LNet: Message forwarding will be enabled in 2s [ 481.567197] LNet: Message forwarding enabled [ 482.070097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 482.096424] LNet: Message forwarding will stop in 2s to 6s [ 486.769567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 487.632195] LNet: Message forwarding disabled [ 489.698660] LNet: Message forwarding will be enabled in 2s [ 492.838947] LNet: Message forwarding enabled [ 496.379752] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 21:02:56 (1763344976) [ 496.511091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.424952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.499231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 502.384213] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 21:03:02 (1763344982) [ 502.519473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.569620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.619749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 502.638156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 502.669681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 502.693735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 502.723849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 502.761548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 502.806688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 502.837545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 502.881747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 502.914115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 502.945643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 502.978512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 503.010859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 503.041818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 503.078328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 503.120904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 503.150499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 503.189283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 503.225725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 503.253788] LNetError: 28857:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 508.825395] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 21:03:08 (1763344988) [ 509.192791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.237500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.330501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.390405] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 21:03:14 (1763344994) [ 517.396471] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg235-server [ 518.903785] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 21:03:18 (1763344998) [ 519.421688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.508904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.593249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 519.653285] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 521.812715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[36-40]@tcp [ 521.825342] LNet: 30573:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 521.890074] LNetError: 30574:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.39@tcp has gone from up to down [ 521.918887] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 522.918204] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 522.937100] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 523.950246] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 523.970598] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 526.009853] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.39@tcp is being used as a gateway but routing feature is not turned on [ 526.015271] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 528.555760] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 21:03:28 (1763345008) [ 530.092361] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.36@tcp is being used as a gateway but routing feature is not turned on [ 530.128371] LNetError: 30362:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 530.568308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg235-server [ 532.512290] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 21:03:31 (1763345011) [ 532.738303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.755652] LNetError: 31508:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 532.771327] LNet: Removed LNI 192.168.202.35@tcp [ 532.801888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.858670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 532.882750] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 533.745483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 533.789900] LNet: 31625:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 533.810986] LNet: 31625:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 533.829904] LNetError: 31512:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.38@tcp has gone from up to down [ 533.839184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[36-41/2]@tcp [ 533.854295] LNetError: 31512:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 539.787909] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 21:03:39 (1763345019) [ 539.957040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.986728] LNetError: 32218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.011540] LNet: Removed LNI 192.168.202.35@tcp [ 540.039339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.153662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 540.218970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.261921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.329435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.404791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.454399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.503818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 540.567829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.619792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.694933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.766195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.810587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.853741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 540.933298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.977246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.052422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.120654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.158857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.197722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 541.260422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.296292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.353300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.425969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.470880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.520353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 541.580717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.612266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.676090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 541.734925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.765994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.815523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 541.848341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.885465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.959120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 548.363042] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 21:03:47 (1763345027) [ 549.941795] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg235-server [ 551.401395] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 21:03:51 (1763345031) [ 552.902505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg235-server [ 554.250543] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 21:03:54 (1763345034) [ 554.856656] Key type .llcrypt unregistered [ 554.859254] Key type ._llcrypt unregistered [ 558.914213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 558.986188] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 558.994396] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 559.001827] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 561.476490] Key type ._llcrypt registered [ 561.478293] Key type .llcrypt registered [ 561.585715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 561.601233] alg: No test for adler32 (adler32-zlib) [ 562.560299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.637069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 562.658117] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 563.237326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 563.254187] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 565.236686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 565.260910] LNet: Removed LNI 10.1.2.3@tcp [ 565.334737] LNet: 34333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 565.346123] LNetError: 34333:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 565.358704] LNet: Removed LNI 192.168.202.35@tcp [ 565.871787] Key type .llcrypt unregistered [ 565.873538] Key type ._llcrypt unregistered [ 575.410343] Key type ._llcrypt registered [ 575.411731] Key type .llcrypt registered [ 576.598593] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 21:04:16 (1763345056) [ 577.061629] Key type .llcrypt unregistered [ 577.063151] Key type ._llcrypt unregistered [ 581.469751] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 581.684114] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 581.687529] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 581.690223] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 584.120241] Key type ._llcrypt registered [ 584.121856] Key type .llcrypt registered [ 584.332821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.349339] alg: No test for adler32 (adler32-zlib) [ 585.310917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.363962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.395527] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 586.035882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 586.058669] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.295720] LNet: 35817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.305426] LNetError: 35817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.316787] LNet: Removed LNI 192.168.202.35@tcp [ 588.326144] LNet: Removed LNI 10.1.2.3@tcp [ 588.776663] Key type .llcrypt unregistered [ 588.778583] Key type ._llcrypt unregistered [ 597.517683] Key type ._llcrypt registered [ 597.521791] Key type .llcrypt registered [ 598.671232] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 21:04:38 (1763345078) [ 599.243524] Key type .llcrypt unregistered [ 599.248255] Key type ._llcrypt unregistered [ 603.814600] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 603.947809] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 603.951721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 603.954429] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 606.483832] Key type ._llcrypt registered [ 606.492962] Key type .llcrypt registered [ 606.687576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.701462] alg: No test for adler32 (adler32-zlib) [ 607.611589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.661596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 607.688826] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 609.309638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 609.324949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 611.093435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 611.108824] LNet: Removed LNI 10.1.2.3@tcp [ 611.127222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 611.151876] LNet: Removed LNI 10.1.2.31@tcp [ 611.180716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 611.211303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 611.732064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 611.751676] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 612.514684] LNet: 37730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.523222] LNetError: 37730:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 612.530488] LNet: Removed LNI 10.1.2.3@tcp [ 612.541763] LNet: Removed LNI 10.1.2.31@tcp [ 613.095682] Key type .llcrypt unregistered [ 613.097385] Key type ._llcrypt unregistered [ 622.868396] Key type ._llcrypt registered [ 622.871814] Key type .llcrypt registered [ 624.000896] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:05:03 (1763345103) [ 624.539456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.552788] alg: No test for adler32 (adler32-zlib) [ 625.458984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.510686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 625.544380] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 626.300590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.319412] LNet: 38678:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 626.326757] LNetError: 38678:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 626.346915] LNet: Removed LNI 192.168.202.35@tcp [ 626.366518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.413310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.35@tcp [ 626.446907] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 627.204432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.35@tcp [ 627.238862] LNet: Removed LNI 192.168.202.35@tcp [ 627.285909] LNet: 38813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.900628] Key type .llcrypt unregistered [ 627.902448] Key type ._llcrypt unregistered [ 637.406429] Key type ._llcrypt registered [ 637.407547] Key type .llcrypt registered [ 638.624497] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:05:18 (1763345118) [ 640.427509] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg235-server [ 642.175749] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:05:21 (1763345121) [ 642.653338] Key type .llcrypt unregistered [ 642.658151] Key type ._llcrypt unregistered [ 647.142374] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 647.217116] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 647.224391] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 647.233723] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 649.908586] Key type ._llcrypt registered [ 649.910185] Key type .llcrypt registered [ 650.112170] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.128995] alg: No test for adler32 (adler32-zlib) [ 651.138886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.191796] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 652.137540] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 654.317783] LNet: 40644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.325178] LNetError: 40644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.333249] LNet: Removed LNI 192.168.202.35@tcp1 [ 654.349316] LNet: Removed LNI 10.1.2.3@tcp2 [ 654.870156] Key type .llcrypt unregistered [ 654.871710] Key type ._llcrypt unregistered [ 664.966072] Key type ._llcrypt registered [ 664.967390] Key type .llcrypt registered [ 666.083457] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:05:45 (1763345145) [ 666.506312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.520483] alg: No test for adler32 (adler32-zlib) [ 667.380714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.447459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.484056] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 668.322683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.351904] LNet: 41609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.362328] LNetError: 41609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.380375] LNet: Removed LNI 192.168.202.35@tcp [ 668.397810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.480378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 668.500116] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 674.303190] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:05:54 (1763345154) [ 674.555866] LNet: 42200:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.560105] LNetError: 42200:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.571196] LNet: Removed LNI 192.168.202.35@tcp [ 675.200512] Key type .llcrypt unregistered [ 675.204271] Key type ._llcrypt unregistered [ 679.805746] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 679.985065] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 680.007966] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 680.018030] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 682.530390] Key type ._llcrypt registered [ 682.532026] Key type .llcrypt registered [ 682.729332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 682.753788] alg: No test for adler32 (adler32-zlib) [ 683.657145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.715469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 683.757616] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 684.463926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 684.482759] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.392128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.403888] LNet: 42938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.408843] LNetError: 42938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.412676] LNet: Removed LNI 192.168.202.35@tcp [ 686.420771] LNet: Removed LNI 10.1.2.3@tcp [ 686.443501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.500577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 686.532937] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 686.538367] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 686.896867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.921763] LNet: 42965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 686.929145] LNetError: 42965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 686.933355] LNetError: 42965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 686.937945] LNet: Removed LNI 192.168.202.35@tcp [ 686.949836] LNet: Removed LNI 10.1.2.3@tcp [ 686.985666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.040836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 687.066856] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 687.072159] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 687.473279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 687.489542] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 688.444495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.461322] LNet: 43127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.467497] LNetError: 43127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 688.472413] LNetError: 43127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 688.478138] LNet: Removed LNI 192.168.202.35@tcp [ 688.480649] LNet: Removed LNI 10.1.2.3@tcp [ 688.488230] LNet: Removed LNI 192.168.202.35@tcp2 [ 688.515448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.569948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 688.594335] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 688.598677] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 688.609502] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 694.479819] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:06:14 (1763345174) [ 694.576258] LNet: 43718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.584982] LNetError: 43718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.588141] LNetError: 43718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 694.608320] LNet: Removed LNI 192.168.202.35@tcp [ 694.611850] LNet: Removed LNI 10.1.2.3@tcp [ 694.630045] LNet: Removed LNI 192.168.202.35@tcp2 [ 695.185728] Key type .llcrypt unregistered [ 695.187193] Key type ._llcrypt unregistered [ 699.947791] Key type ._llcrypt registered [ 699.949992] Key type .llcrypt registered [ 700.149271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 700.168709] alg: No test for adler32 (adler32-zlib) [ 701.042622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.097413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 701.123514] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 701.954545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.967904] LNet: 44116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.973114] LNetError: 44116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.981867] LNet: Removed LNI 192.168.202.35@tcp [ 702.011630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.084543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 702.101280] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 702.458717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.480270] LNet: 44146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.490497] LNetError: 44146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 702.503513] LNet: Removed LNI 192.168.202.35@tcp [ 702.525610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.595706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 702.618719] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 702.986095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.024724] LNet: Removed LNI 192.168.202.35@tcp [ 703.045621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.098918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 703.124606] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 703.651152] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 703.842681] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 703.844870] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 703.846922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 706.149667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.166396] LNet: 44254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.170215] LNet: 44254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 706.176601] LNetError: 44254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.182442] LNetError: 44254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 706.194704] LNet: Removed LNI 192.168.202.35@tcp [ 706.214169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.272855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 706.291023] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 711.518714] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:06:31 (1763345191) [ 711.675672] LNet: 44851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.688730] LNetError: 44851:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.711568] LNet: Removed LNI 192.168.202.35@tcp [ 712.278869] Key type .llcrypt unregistered [ 712.284530] Key type ._llcrypt unregistered [ 717.044103] Key type ._llcrypt registered [ 717.062711] Key type .llcrypt registered [ 717.282907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 717.294476] alg: No test for adler32 (adler32-zlib) [ 718.198963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.257850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 718.281554] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 719.133150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.157373] LNet: 45245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.196255] LNetError: 45245:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.221556] LNet: Removed LNI 192.168.202.35@tcp [ 719.253047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.309251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 719.365085] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 725.094903] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:06:44 (1763345204) [ 725.328971] LNet: 45837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.332883] LNetError: 45837:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.353367] LNet: Removed LNI 192.168.202.35@tcp [ 725.921822] Key type .llcrypt unregistered [ 725.925368] Key type ._llcrypt unregistered [ 730.963730] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 731.081461] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 731.084413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 731.087455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 733.554938] Key type ._llcrypt registered [ 733.556788] Key type .llcrypt registered [ 733.747245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 733.769106] alg: No test for adler32 (adler32-zlib) [ 734.674963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.718636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 734.740912] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 735.690932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 735.716444] LNet: 46284:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 735.722501] LNetError: 46284:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.733901] LNet: Removed LNI 192.168.202.35@tcp [ 735.756821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.821533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 735.847270] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 741.815483] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:07:01 (1763345221) [ 742.011138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 742.041138] LNet: 46883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.047213] LNetError: 46883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 742.066932] LNet: Removed LNI 192.168.202.35@tcp [ 742.093466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.211379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 748.455841] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:07:07 (1763345227) [ 748.595243] LNet: 47457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.252853] Key type .llcrypt unregistered [ 749.254684] Key type ._llcrypt unregistered [ 754.936121] Key type ._llcrypt registered [ 754.941074] Key type .llcrypt registered [ 755.119620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.132303] alg: No test for adler32 (adler32-zlib) [ 756.020970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.072938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.101946] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 757.058897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.077849] LNet: 47858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.087534] LNetError: 47858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.102447] LNet: Removed LNI 192.168.202.35@tcp [ 757.120333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.246419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 757.276297] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 763.869839] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:07:23 (1763345243) [ 764.280552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 764.296504] LNet: 48451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 764.315871] LNetError: 48451:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 764.337385] LNet: Removed LNI 192.168.202.35@tcp [ 764.375906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 764.477400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 769.868145] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:07:29 (1763345249) [ 771.293660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 772.845541] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:07:32 (1763345252) [ 774.299478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 775.712969] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:07:35 (1763345255) [ 777.258428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 778.570989] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:07:38 (1763345258) [ 779.913929] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 781.381678] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:07:41 (1763345261) [ 782.669641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 783.968482] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:07:43 (1763345263) [ 784.083130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.116224] LNet: 50503:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.135515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.184203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.221932] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 784.943908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.35 [ 784.964165] LNet: Removed LNI 192.168.202.35@tcp [ 790.854726] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:07:50 (1763345270) [ 791.107132] LNet: 51207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 791.114397] LNetError: 51207:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 791.125841] LNet: Removed LNI 192.168.202.35@tcp [ 791.576968] Key type .llcrypt unregistered [ 791.578582] Key type ._llcrypt unregistered [ 795.769681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 796.022314] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 796.029539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 796.033982] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.563339] Key type ._llcrypt registered [ 798.565229] Key type .llcrypt registered [ 798.714305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.726372] alg: No test for adler32 (adler32-zlib) [ 799.904206] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 805.472680] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:08:05 (1763345285) [ 805.664425] LNet: 52103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 805.670189] LNetError: 52103:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 805.697261] LNet: Removed LNI 10.1.2.3@tcp [ 806.179542] Key type .llcrypt unregistered [ 806.181774] Key type ._llcrypt unregistered [ 810.799605] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.061213] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.069187] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.077851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 812.645302] Key type ._llcrypt registered [ 812.647015] Key type .llcrypt registered [ 812.790863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 812.803173] alg: No test for adler32 (adler32-zlib) [ 813.770442] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.106837] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:08:19 (1763345299) [ 820.489152] LNet: 52987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.500633] LNetError: 52987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 820.514738] LNet: Removed LNI 10.1.2.3@tcp [ 821.035454] Key type .llcrypt unregistered [ 821.037821] Key type ._llcrypt unregistered [ 825.936467] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 826.220207] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.230295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.241430] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 828.102598] Key type ._llcrypt registered [ 828.104145] Key type .llcrypt registered [ 828.292387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 828.315517] alg: No test for adler32 (adler32-zlib) [ 829.276272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 835.861902] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:08:35 (1763345315) [ 836.086184] LNet: 53868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 836.092683] LNetError: 53868:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 836.106859] LNet: Removed LNI 10.1.2.3@tcp [ 836.641634] Key type .llcrypt unregistered [ 836.645122] Key type ._llcrypt unregistered [ 842.544345] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.858790] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.861337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.863894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.641414] Key type ._llcrypt registered [ 845.659475] Key type .llcrypt registered [ 845.886719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 845.897627] alg: No test for adler32 (adler32-zlib) [ 846.788428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.890104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 856.274549] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:08:55 (1763345335) [ 856.875447] LNet: 55095:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 856.884986] LNetError: 55095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 856.906424] LNet: Removed LNI 10.1.2.3@tcp [ 857.650608] Key type .llcrypt unregistered [ 857.657912] Key type ._llcrypt unregistered [ 864.246520] Key type ._llcrypt registered [ 864.248266] Key type .llcrypt registered [ 864.512552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.542956] alg: No test for adler32 (adler32-zlib) [ 865.427649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 865.467123] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 872.135055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 872.952744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 873.008928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 873.052429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 873.075025] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 873.339769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 873.375540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 873.391548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 873.430327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 873.455191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 873.487281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 873.538678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 873.575370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 873.600319] LNet: 55529:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 873.620676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 873.660361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 873.685040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 873.695407] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 873.710153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 873.731071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 873.759374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 873.790328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 874.293059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 874.331303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 874.381789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 874.409765] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 874.697843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 874.732857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 874.760485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 874.800568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 874.826637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 874.858123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 874.883140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 874.903837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 874.921476] LNet: 55638:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 874.933913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 874.967042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 874.992472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 875.022081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 875.053709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 875.116918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 875.171931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 875.686699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 875.736242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 875.782183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 875.799342] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 875.802692] LNet: Skipped 1 previous similar message [ 875.987845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 876.009268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 876.050398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 876.097227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 876.131839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 876.166584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 876.191500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 876.223314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 876.242996] LNet: 55747:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 876.265823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 876.317735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 876.358412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 876.381381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 876.411433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 876.442508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 876.464566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 876.872848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 876.897489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 876.922275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 877.111927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 877.133681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 877.164299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 877.200602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 877.265260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 877.305414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 877.331927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 877.360958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 877.377375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 877.401763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 877.439829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 877.498656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 877.527279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 877.563744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 877.600227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 878.023697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 878.055531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 878.096403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 878.112935] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 878.116338] LNet: Skipped 1 previous similar message [ 878.393355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 878.437368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 878.469290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 878.493850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 878.538499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 878.581813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 878.605631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 878.635611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 878.655240] LNet: 55965:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 878.661099] LNet: 55965:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 878.676995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 878.710972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 878.748813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 878.787407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 878.807762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 878.842848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 878.868605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 879.304927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 879.331180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 879.356379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 879.563923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 879.595682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 879.626350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 879.658318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 879.680695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 879.717668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 879.757558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 879.811267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 879.855802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 879.899532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 879.924656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 879.950802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 879.993409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 880.035405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 880.068576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 880.294785] LNet: 56115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 880.299313] LNetError: 56115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.319710] LNet: Removed LNI 192.168.202.35@tcp [ 881.620551] Key type .llcrypt unregistered [ 881.623165] Key type ._llcrypt unregistered [ 895.745396] Key type ._llcrypt registered [ 895.746988] Key type .llcrypt registered [ 897.125241] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:09:36 (1763345376) [ 898.044827] Key type .llcrypt unregistered [ 898.046514] Key type ._llcrypt unregistered [ 903.848151] Key type ._llcrypt registered [ 903.849907] Key type .llcrypt registered [ 904.093542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 904.115229] alg: No test for adler32 (adler32-zlib) [ 905.075910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 905.096759] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 911.817619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 920.628572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.659751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 920.697341] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 921.991851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 922.036060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 922.077724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 922.112850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 922.149421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 1 [ 922.180565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 1 [ 922.211393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 1 [ 922.243769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 1 [ 922.289449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 922.310672] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 922.322454] LNet: 2 local NIs in recovery (showing 2): 192.168.202.35@tcp, 192.168.202.35@tcp1 [ 922.542741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 922.575883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 922.614756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 922.655274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 922.715478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 922.756434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 922.787401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 922.808143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 922.820872] LNet: 57575:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 922.835927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 922.857273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 922.880382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 922.908566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 922.934313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 922.953424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 922.972315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 923.460775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 923.484000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 923.521476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 923.569898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 923.596435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 1 [ 923.638973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 1 [ 923.664450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 1 [ 923.695567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 1 [ 923.725541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 923.754708] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 923.758266] LNet: Skipped 2 previous similar messages [ 923.944783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 923.985516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 924.024131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 924.048326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 924.086350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 924.116223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 924.148985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 924.195653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 924.211530] LNet: 57696:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 924.224214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 924.248611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 924.283872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 924.306850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 924.356966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 924.409603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 924.440711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 924.928296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 924.976211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 925.011470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 925.042105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 925.086479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 1 [ 925.116677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 1 [ 925.162922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 1 [ 925.199554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 1 [ 925.228227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 925.249121] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 925.259253] LNet: Skipped 2 previous similar messages [ 925.448921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 925.473742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 925.496551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 925.527258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 925.547230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 925.582622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 925.609785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 925.629833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 925.644253] LNet: 57817:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 925.656601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 925.676230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 925.702543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 925.729792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 925.754262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 925.781604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 925.806331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 926.250209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 926.272186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 926.312226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 926.338561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 926.381708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 1 [ 926.431121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 1 [ 926.464118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 1 [ 926.512854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 1 [ 926.550873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 926.864911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 926.888480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 926.917461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 926.953500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 926.982449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 927.001166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 927.030170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 927.063754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 927.098092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 927.141536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 927.185074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 927.214627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 927.249849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 927.286924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.336788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 927.772301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 927.807936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 927.855300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 927.887859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 927.934654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 1 [ 927.990515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 1 [ 928.048411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 1 [ 928.092535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 1 [ 928.142788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 928.164500] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 928.169836] LNet: Skipped 5 previous similar messages [ 928.332572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 928.361748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 928.391934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 928.423482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 928.453105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 928.478289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 928.506116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 928.533391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 928.548554] LNet: 58060:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.553229] LNet: 58060:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 928.565454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 928.590369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 928.619802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 928.649495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 928.672758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 928.690931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 928.731680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 929.185341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 929.220380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 929.251968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 929.283412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 929.317533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_error -r 1 [ 929.351539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_error -r 1 [ 929.385132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_error -r 1 [ 929.425141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_error -r 1 [ 929.470706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 929.679180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.702521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.726485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.742860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.770316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.799420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.838142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.880888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.923915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.953931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.977530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.011502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.039677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.071907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.097685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 934.362416] LNet: 58256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.379505] LNetError: 58256:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.384358] LNet: Removed LNI 192.168.202.35@tcp [ 934.402827] LNet: Removed LNI 192.168.202.35@tcp1 [ 934.766608] Key type .llcrypt unregistered [ 934.771305] Key type ._llcrypt unregistered [ 949.048200] Key type ._llcrypt registered [ 949.053021] Key type .llcrypt registered [ 950.464567] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:10:29 (1763345429) [ 951.400205] Key type .llcrypt unregistered [ 951.403017] Key type ._llcrypt unregistered [ 957.113539] Key type ._llcrypt registered [ 957.115295] Key type .llcrypt registered [ 957.498469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 957.540329] alg: No test for adler32 (adler32-zlib) [ 958.525650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 958.569128] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 965.815442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 966.633899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 966.683313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 966.730706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 966.758874] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 967.165921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 967.187591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 967.225807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 967.298607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 967.343887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 967.387099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 967.409383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 967.430896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 967.467369] LNet: 59496:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 967.512992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 967.549518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 967.632728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 967.655490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 967.692752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 967.761411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 967.838457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 968.446176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 968.472146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 968.501832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 968.522387] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 968.693105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 968.734865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 968.768245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 968.785215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 968.828835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 968.880611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 968.917486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 968.943408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 968.961663] LNet: 59606:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 968.983592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 969.054204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 969.096469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 969.140156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.179788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 969.213256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 969.259301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 969.921116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 969.955894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 969.993970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 970.028363] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 970.305213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 970.342456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 970.380892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 970.420728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 970.453783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 970.484640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 970.515906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 970.543779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 970.569303] LNet: 59716:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.601224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 970.668561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 970.707990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 970.731740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 970.803286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.868170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 970.917584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 971.328261] LNet: 59756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.350585] LNetError: 59756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.380736] LNet: Removed LNI 192.168.202.35@tcp [ 971.843758] Key type .llcrypt unregistered [ 971.853388] Key type ._llcrypt unregistered [ 988.601025] Key type ._llcrypt registered [ 988.602354] Key type .llcrypt registered [ 989.854700] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:11:09 (1763345469) [ 990.757313] Key type .llcrypt unregistered [ 990.762840] Key type ._llcrypt unregistered [ 998.746075] Key type ._llcrypt registered [ 998.747488] Key type .llcrypt registered [ 999.011417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 999.026298] alg: No test for adler32 (adler32-zlib) [ 999.933194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 999.956798] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1009.202797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1018.928343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.991467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1019.036294] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1020.725992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1020.755192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1020.786329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1020.838876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1020.913263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 1 [ 1020.983788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 1 [ 1021.058973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 1 [ 1021.081111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 1 [ 1021.141914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1021.161807] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1021.167375] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1021.392598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1021.432480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1021.466958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1021.498947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1021.526323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1021.569512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1021.596237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1021.636419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1021.654031] LNet: 61215:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1021.685511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1021.722770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1021.775386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1021.814748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1021.846352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1021.893895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1021.937139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1021.974881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1022.424708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1022.451711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1022.487079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1022.532920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1022.592223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_error -r 1 [ 1022.654376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_error -r 1 [ 1022.706790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_error -r 1 [ 1022.767960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_error -r 1 [ 1022.830029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1022.856438] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1022.865028] LNet: Skipped 2 previous similar messages [ 1023.071397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1023.100733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1023.128498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1023.165540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1023.215200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1023.242505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1023.284196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1023.319871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1023.334592] LNet: 61338:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1023.350683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1023.385875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1023.413712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1023.456461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1023.501085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1023.534665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1023.569117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1023.595300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1024.093262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1024.127730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1024.151682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1024.175952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1024.216968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_timeout -r 1 [ 1024.272668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_timeout -r 1 [ 1024.312609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_timeout -r 1 [ 1024.341489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_timeout -r 1 [ 1024.373033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.135@tcp [ 1024.386469] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1024.391153] LNet: Skipped 1 previous similar message [ 1024.552710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1024.588809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1024.612215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1024.650939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1024.685890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1024.745609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1024.786753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1024.838859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1024.855994] LNet: 61462:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1024.881538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1024.906210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1024.930905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1024.963823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1024.995771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.022412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.042273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1025.063707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.969885] LNet: 61538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.979672] LNetError: 61538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1028.988884] LNet: Removed LNI 192.168.202.35@tcp [ 1029.008578] LNet: Removed LNI 192.168.202.35@tcp1 [ 1029.411846] Key type .llcrypt unregistered [ 1029.413389] Key type ._llcrypt unregistered [ 1045.154034] Key type ._llcrypt registered [ 1045.156915] Key type .llcrypt registered [ 1046.782247] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:12:05 (1763345525) [ 1047.531445] Key type .llcrypt unregistered [ 1047.541588] Key type ._llcrypt unregistered [ 1054.020406] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1054.227884] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1054.234138] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1054.237599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1056.852770] Key type ._llcrypt registered [ 1056.854428] Key type .llcrypt registered [ 1057.001023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1057.018169] alg: No test for adler32 (adler32-zlib) [ 1057.986943] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1058.041919] LNet: 62627:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.049393] LNetError: 62627:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1058.058742] LNet: Removed LNI 192.168.202.35@tcp [ 1058.876448] Key type .llcrypt unregistered [ 1058.879560] Key type ._llcrypt unregistered [ 1064.745913] Key type ._llcrypt registered [ 1064.751730] Key type .llcrypt registered [ 1064.949840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1064.968322] alg: No test for adler32 (adler32-zlib) [ 1065.848207] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1065.850586] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1065.884839] LNet: 62896:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1065.891532] LNetError: 62896:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1065.896875] LNet: Removed LNI 192.168.202.35@tcp [ 1065.911163] LNet: Removed LNI 10.1.2.3@tcp1 [ 1066.631624] Key type .llcrypt unregistered [ 1066.634760] Key type ._llcrypt unregistered [ 1072.566419] Key type ._llcrypt registered [ 1072.576196] Key type .llcrypt registered [ 1072.741962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1072.766362] alg: No test for adler32 (adler32-zlib) [ 1073.678107] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1073.679462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1073.701951] LNet: 63165:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1073.706759] LNetError: 63165:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1073.709928] LNet: Removed LNI 192.168.202.35@tcp [ 1073.730759] LNet: Removed LNI 10.1.2.3@tcp [ 1074.526728] Key type .llcrypt unregistered [ 1074.528648] Key type ._llcrypt unregistered [ 1079.748946] Key type ._llcrypt registered [ 1079.751790] Key type .llcrypt registered [ 1079.934167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1079.952805] alg: No test for adler32 (adler32-zlib) [ 1080.903176] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1080.905549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1080.930766] LNet: 63435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.936679] LNetError: 63435:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.940115] LNet: Removed LNI 192.168.202.35@tcp [ 1080.957149] LNet: Removed LNI 10.1.2.3@tcp [ 1081.651908] Key type .llcrypt unregistered [ 1081.658822] Key type ._llcrypt unregistered [ 1087.804459] Key type ._llcrypt registered [ 1087.805846] Key type .llcrypt registered [ 1088.060704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1088.080488] alg: No test for adler32 (adler32-zlib) [ 1088.993836] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1088.998748] LNetError: 63695:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1089.004297] LNetError: Error -100 starting up LNI tcp [ 1089.015862] LNet: Removed LNI 192.168.202.35@tcp [ 1089.698602] Key type .llcrypt unregistered [ 1089.700613] Key type ._llcrypt unregistered [ 1100.442774] Key type ._llcrypt registered [ 1100.444610] Key type .llcrypt registered [ 1101.533393] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:13:01 (1763345581) [ 1102.524456] Key type .llcrypt unregistered [ 1102.528597] Key type ._llcrypt unregistered [ 1108.704711] Key type ._llcrypt registered [ 1108.715036] Key type .llcrypt registered [ 1108.936392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.950416] alg: No test for adler32 (adler32-zlib) [ 1109.924072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1109.982523] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1117.262899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1117.988755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1118.021503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1118.059543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1118.080668] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1118.189375] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1118.321725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.361257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1118.385713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.416072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.444719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.474772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.505755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.534361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.552898] LNet: 64946:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.571909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.599300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.627586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.668423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.701817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.728070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.966753] LNet: 64984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.973206] LNetError: 64984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.994134] LNet: Removed LNI 192.168.202.35@tcp [ 1119.467865] Key type .llcrypt unregistered [ 1119.469934] Key type ._llcrypt unregistered [ 1135.390092] Key type ._llcrypt registered [ 1135.391783] Key type .llcrypt registered [ 1135.538574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1135.557836] alg: No test for adler32 (adler32-zlib) [ 1136.502836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1136.539369] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1142.815403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1150.020689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1150.098082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1150.122971] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1151.426714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1151.453510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1151.475598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1151.510946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1151.571644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e network_timeout -r 1 [ 1151.617822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e network_timeout -r 1 [ 1151.673370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e network_timeout -r 1 [ 1151.727671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e network_timeout -r 1 [ 1151.772413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1151.799346] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1151.908958] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1151.912093] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1152.024552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1152.071414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1152.110932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1152.151205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1152.200920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1152.254149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1152.301915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1152.363918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1152.389674] LNet: 65855:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1152.422705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.464371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.489314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.512815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.545160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.579554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.322267] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1156.568089] LNet: 65928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.572239] LNetError: 65928:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.583682] LNet: Removed LNI 192.168.202.35@tcp [ 1156.594762] LNet: Removed LNI 192.168.202.35@tcp1 [ 1157.002697] Key type .llcrypt unregistered [ 1157.005081] Key type ._llcrypt unregistered [ 1169.780495] Key type ._llcrypt registered [ 1169.782154] Key type .llcrypt registered [ 1171.097082] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:14:10 (1763345650) [ 1171.528643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.547909] alg: No test for adler32 (adler32-zlib) [ 1172.481427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.533606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1172.557132] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1173.240993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1173.259937] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1174.020177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1174.098770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1174.197272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1174.263858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1174.321599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1174.369632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1174.445029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1174.484850] LNetError: 67023:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1174.567461] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1174.569221] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1176.609611] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1177.043392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1180.703542] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1181.598390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1181.649517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.702842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1181.715345] LNet: 67130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1181.720108] LNetError: 67130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.751101] LNet: Removed LNI 192.168.202.35@tcp [ 1182.760482] LNetError: 67130:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.780467] LNet: Removed LNI 192.168.202.35@tcp1 [ 1182.808738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.927545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1182.967551] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1183.665811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1183.685665] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1184.460720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1184.502220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.544913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1184.595556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1184.637083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1184.659247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1184.699358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1184.719945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1184.734050] LNetError: 67403:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1184.931584] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1187.151979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1190.047340] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1190.055540] LNet: Skipped 1 previous similar message [ 1195.039299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1195.070659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1195.121197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1201.531688] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:14:41 (1763345681) [ 1201.750718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1201.767900] LNet: 68109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1201.773111] LNetError: 68109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.779470] LNet: Removed LNI 192.168.202.35@tcp [ 1201.793464] LNet: Removed LNI 192.168.202.35@tcp1 [ 1201.822624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1201.888976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1201.928865] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1202.676950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1202.689972] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1203.555668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1203.614691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1203.650703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1203.682746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1203.715389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1203.753433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1203.784713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1203.887348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1203.935345] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.35@tcp [ 1204.960469] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1204.979081] LNet: Skipped 1 previous similar message [ 1220.238447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1220.375431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1220.407905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1220.435556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 500 [ 1220.521655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1225.996625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1226.038356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1226.071134] LNet: 68601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1226.075816] LNetError: 68601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1226.080179] LNetError: 68601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1226.086815] LNet: Removed LNI 192.168.202.35@tcp [ 1226.100435] LNet: Removed LNI 192.168.202.35@tcp1 [ 1226.127955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1226.202661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1226.245835] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1226.970541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1226.988596] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1227.705184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1227.737394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.799676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1227.830980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1227.857070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1227.877517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1227.910324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1227.938600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.35@tcp --health 0 [ 1228.193329] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1228.202378] LNet: Skipped 3 previous similar messages [ 1229.237745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1239.269837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1239.302856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.341807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.344592] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:15:25 (1763345725) [ 1246.764481] LNet: 69647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1246.770232] LNetError: 69647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1246.775173] LNetError: 69647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1246.779527] LNet: Removed LNI 192.168.202.35@tcp [ 1246.788634] LNet: Removed LNI 192.168.202.35@tcp1 [ 1247.363101] Key type .llcrypt unregistered [ 1247.364636] Key type ._llcrypt unregistered [ 1252.961479] Key type ._llcrypt registered [ 1252.963058] Key type .llcrypt registered [ 1253.185425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1253.200902] alg: No test for adler32 (adler32-zlib) [ 1254.089973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1254.118441] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1262.136286] Lustre: DEBUG MARKER: Initial discovery [ 1262.396230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1264.580806] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1264.646038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1264.720449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1264.754450] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1264.780710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1264.826852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -l 3 [ 1264.865698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.202.135@tcp -r 1 -m GET -e local_error [ 1264.962307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.135@tcp -r 1 -m PUT -l 6 [ 1265.377850] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1266.528966] Lustre: DEBUG MARKER: Force oleg235-client.virtnet to discover 192.168.202.135@tcp (in background) [ 1266.591138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.135@tcp [ 1267.909410] Lustre: DEBUG MARKER: Force oleg235-server to discover 192.168.202.35@tcp [ 1269.672878] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1270.307175] Lustre: DEBUG MARKER: Wait for 70396 [ 1271.797609] Lustre: DEBUG MARKER: Finished wait on 70396 [ 1271.931030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1271.982614] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1272.014745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1272.437552] LNet: 70854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.443227] LNetError: 70854:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.461484] LNet: Removed LNI 192.168.202.35@tcp [ 1273.006787] Key type .llcrypt unregistered [ 1273.008908] Key type ._llcrypt unregistered [ 1286.408218] Key type ._llcrypt registered [ 1286.409548] Key type .llcrypt registered [ 1287.648117] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:16:07 (1763345767) [ 1288.208769] Key type .llcrypt unregistered [ 1288.211070] Key type ._llcrypt unregistered [ 1293.657196] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1293.847160] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1293.850165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1293.861601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1296.603807] Key type ._llcrypt registered [ 1296.607917] Key type .llcrypt registered [ 1296.857051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1296.875682] alg: No test for adler32 (adler32-zlib) [ 1297.757263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.799619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.816595] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1298.511541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1298.540755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1306.802437] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:16:26 (1763345786) [ 1306.996371] LNet: 72919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1307.003438] LNetError: 72919:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1307.008234] LNet: Removed LNI 192.168.202.35@tcp [ 1307.019811] LNet: Removed LNI 10.1.2.3@tcp [ 1307.814852] Key type .llcrypt unregistered [ 1307.816770] Key type ._llcrypt unregistered [ 1313.983181] Key type ._llcrypt registered [ 1313.984909] Key type .llcrypt registered [ 1314.206463] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1314.219846] alg: No test for adler32 (adler32-zlib) [ 1315.166329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.239909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1315.285654] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1316.316724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1316.344170] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1317.502365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.35@tcp [ 1318.561413] LNetError: Unexpected error -22 connecting to 192.168.202.35@tcp at host 192.168.202.35:988 [ 1325.651841] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:16:44 (1763345804) [ 1325.827876] LNet: 74043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.850822] LNetError: 74043:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.885869] LNet: Removed LNI 192.168.202.35@tcp [ 1325.948859] LNet: Removed LNI 10.1.2.3@tcp [ 1326.906455] Key type .llcrypt unregistered [ 1326.913875] Key type ._llcrypt unregistered [ 1334.047991] Key type ._llcrypt registered [ 1334.052963] Key type .llcrypt registered [ 1334.243916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1334.255616] alg: No test for adler32 (adler32-zlib) [ 1335.185773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.229869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1335.244940] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1335.769369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1335.786822] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1336.754349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.35@tcp1 --nid 192.168.202.35@tcp2 [ 1341.649540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp2 [ 1341.688500] LNetError: 74883:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.35@tcp1 to 192.168.202.35@tcp2 - Route table may be misconfigured [ 1341.724507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp2 192.168.202.35@tcp1 [ 1348.132721] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:17:07 (1763345827) [ 1348.576830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1348.599816] LNet: 75456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1348.616961] LNetError: 75456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1348.623774] LNet: Removed LNI 192.168.202.35@tcp1 [ 1349.674732] LNet: Removed LNI 192.168.202.35@tcp2 [ 1349.693485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.738601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.756257] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1350.633855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1350.651484] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1351.415526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1351.456567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.35@tcp [ 1351.492248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp1 192.168.202.35@tcp1 [ 1351.551603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp -e network_timeout [ 1351.607675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -e network_timeout [ 1351.679144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -e network_timeout [ 1351.724409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -e network_timeout [ 1351.777190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1351.798279] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1351.830082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1351.946103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1352.095651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1352.816901] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1358.696158] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:17:18 (1763345838) [ 1358.871622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.894179] LNet: 76318:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1358.908417] LNetError: 76318:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.915200] LNetError: 76318:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1358.926443] LNet: Removed LNI 192.168.202.35@tcp [ 1359.976464] LNet: Removed LNI 192.168.202.35@tcp1 [ 1360.015909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1360.097467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1360.312882] LNet: 76355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.707454] Key type .llcrypt unregistered [ 1360.708813] Key type ._llcrypt unregistered [ 1373.175985] Key type ._llcrypt registered [ 1373.177961] Key type .llcrypt registered [ 1374.504248] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:17:33 (1763345853) [ 1374.970179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.986777] alg: No test for adler32 (adler32-zlib) [ 1375.840214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1377.016328] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1378.478436] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:17:38 (1763345858) [ 1378.758261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.785220] LNet: 77449:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.828682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.950620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.971456] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1379.688118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1379.709233] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1380.583697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1380.617484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp1 [ 1380.649602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp1 [ 1386.649631] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:17:46 (1763345866) [ 1388.404381] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg235-server [ 1390.171525] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:17:49 (1763345869) [ 1391.946777] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg235-server [ 1393.466763] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:17:53 (1763345873) [ 1395.319367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg235-server [ 1396.976924] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:17:56 (1763345876) [ 1399.156899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg235-server [ 1401.051836] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:18:00 (1763345880) [ 1403.004516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg235-server [ 1404.994750] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:18:04 (1763345884) [ 1407.097758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg235-server [ 1408.826794] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:18:08 (1763345888) [ 1410.706372] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg235-server [ 1412.266369] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:18:11 (1763345891) [ 1413.997641] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg235-server [ 1415.656945] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:18:15 (1763345895) [ 1417.425720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg235-server [ 1419.079938] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:18:18 (1763345898) [ 1419.479110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1419.520536] LNet: 81327:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1419.535509] LNetError: 81327:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1419.544047] LNet: Removed LNI 192.168.202.35@tcp [ 1420.583309] LNet: Removed LNI 192.168.202.35@tcp1 [ 1420.668934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1420.787716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1420.830364] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1421.808985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1421.896298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1422.142216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.160878] LNet: 81475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.165653] LNetError: 81475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.169024] LNetError: 81475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1422.190503] LNet: Removed LNI 192.168.202.35@tcp [ 1422.226427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.288831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1422.313213] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1423.047917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1423.292552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.328371] LNetError: 81621:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1424.360461] LNet: Removed LNI 192.168.202.35@tcp [ 1424.388672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1424.487817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1424.514656] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1425.236510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1425.320778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1426.673924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1426.709482] LNet: 81777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1426.723438] LNet: 81777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1426.738026] LNetError: 81777:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1427.795703] LNet: Removed LNI 192.168.202.35@tcp [ 1427.839815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.933273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1427.995993] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1428.787397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1429.054208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.118758] LNet: Removed LNI 192.168.202.35@tcp [ 1430.161119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.296285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1430.357532] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1431.051515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1431.142234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1431.370581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.397753] LNet: 82070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.402407] LNet: 82070:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1431.410053] LNetError: 82070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1431.417274] LNetError: 82070:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1431.438905] LNet: Removed LNI 192.168.202.35@tcp [ 1431.465685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.579869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1431.638342] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1432.743964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1433.066995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.151970] LNet: Removed LNI 192.168.202.35@tcp [ 1434.238570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.317383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1434.368920] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1435.271181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1435.372060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1435.680979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.748527] LNet: Removed LNI 192.168.202.35@tcp [ 1435.803924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.903392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1435.931825] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1437.134945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1437.395580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.439387] LNet: Removed LNI 192.168.202.35@tcp [ 1438.505191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.582983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1438.633871] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1439.506425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1439.593611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1439.890673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1439.911699] LNet: 82656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.916209] LNet: 82656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1439.923322] LNetError: 82656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.931981] LNetError: 82656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1439.956814] LNet: Removed LNI 192.168.202.35@tcp [ 1439.970950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1440.059648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1440.094569] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1440.979915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1441.285736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1442.342159] LNet: Removed LNI 192.168.202.35@tcp [ 1442.380453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.443974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.478233] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1443.306939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1443.379024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1443.686420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.779897] LNet: Removed LNI 192.168.202.35@tcp [ 1444.810705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.906028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1444.924203] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1445.908666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1447.515602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.555434] LNet: Removed LNI 192.168.202.35@tcp [ 1448.587310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.693718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.744761] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1449.643809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1449.724830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1449.762342] LNet: Removed LNI 192.168.202.35@tcp [ 1449.778035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.868800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1449.896481] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1450.868742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1457.999531] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:18:57 (1763345937) [ 1458.163165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1458.193804] LNet: 83938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.205734] LNet: 83938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1458.230530] LNetError: 83938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.233692] LNetError: 83938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1458.274169] LNet: Removed LNI 192.168.202.35@tcp [ 1458.290743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1458.380500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1458.424197] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1459.480653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1459.509585] LNet: Removed LNI 192.168.202.35@tcp231 [ 1459.544442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1459.628277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1459.699938] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1459.714561] LNet: 84087:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1460.303488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1460.350675] LNet: Removed LNI 192.168.202.35@tcp231 [ 1460.366942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1460.389421] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1461.805737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.835595] LNet: Removed LNI 192.168.202.35@tcp231 [ 1461.863360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.936339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1461.972205] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1461.986068] LNet: 84240:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1462.492694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1462.560319] LNet: Removed LNI 192.168.202.35@tcp231 [ 1462.580574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1462.613962] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/0] [ 1463.852985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.883416] LNet: Removed LNI 192.168.202.35@tcp231 [ 1463.962920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.082722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1464.136911] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1464.147897] LNet: 84391:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1464.622348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1464.657166] LNet: Removed LNI 192.168.202.35@tcp231 [ 1464.693668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1464.737364] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/60] [ 1465.904534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1465.949541] LNet: Removed LNI 192.168.202.35@tcp231 [ 1465.973089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.076305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1466.118374] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1466.563841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1466.605063] LNet: Removed LNI 192.168.202.35@tcp231 [ 1466.623931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1466.647615] LNet: Added LNI 192.168.202.35@tcp231 [8/256/0/180] [ 1473.880342] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:19:13 (1763345953) [ 1473.996273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.050234] LNet: Removed LNI 192.168.202.35@tcp231 [ 1474.076201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1474.140858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1474.168503] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1474.782189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1474.806139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1474.831206] LNet: Removed LNI 192.168.202.35@tcp232 [ 1474.846884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1474.873097] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1475.393660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1475.427615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1475.457859] LNet: Removed LNI 192.168.202.35@tcp232 [ 1475.475811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1475.499769] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1476.156334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1476.189735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1476.207354] LNet: Removed LNI 192.168.202.35@tcp232 [ 1476.227772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1476.247568] LNet: Added LNI 192.168.202.35@tcp232 [8/256/0/180] [ 1476.921945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1476.975949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1477.019951] LNet: Removed LNI 192.168.202.35@tcp232 [ 1477.035988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1482.805881] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:19:22 (1763345962) [ 1484.133600] Key type .llcrypt unregistered [ 1484.135936] Key type ._llcrypt unregistered [ 1490.784985] Key type ._llcrypt registered [ 1490.786691] Key type .llcrypt registered [ 1491.065982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1491.101387] alg: No test for adler32 (adler32-zlib) [ 1492.134591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1492.190834] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1499.499538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1506.450753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1506.508150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1506.556641] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1508.039846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 1508.109575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 1508.171593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 1508.242935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 1508.295382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_interrupt -r 2 [ 1508.336987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_interrupt -r 2 [ 1508.386224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_interrupt -r 2 [ 1508.420844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_interrupt -r 2 [ 1508.449640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1508.474150] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1508.484186] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1508.715950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1508.740526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1508.767286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1508.809826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1508.847547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1508.887733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1508.934984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1508.965859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1508.986511] LNet: 87040:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1509.002827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1509.026802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1509.050332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1509.088051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1509.113072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1509.151657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1509.689423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 1509.722838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 1509.757504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 1509.809750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 1509.846921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_dropped -r 2 [ 1509.885303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_dropped -r 2 [ 1509.922438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_dropped -r 2 [ 1509.961219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_dropped -r 2 [ 1510.021613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1510.061850] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1510.344313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1510.382306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1510.406041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1510.438538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1510.482359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1510.536871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1510.588946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1510.634974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1510.651420] LNet: 87161:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1510.665959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1510.692854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1510.719912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1510.753939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1510.782598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1510.824534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1511.429571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 1511.467088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 1511.528958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 1511.563920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 1511.584615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_aborted -r 2 [ 1511.614693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_aborted -r 2 [ 1511.685509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_aborted -r 2 [ 1511.734065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_aborted -r 2 [ 1511.791590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1511.813209] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1512.104736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1512.170136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1512.187925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1512.220707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1512.251597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1512.282624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1512.318927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1512.369185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1512.400955] LNet: 87281:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1512.419834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1512.440807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1512.463387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.505511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.551755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1512.585369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1513.072331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 1513.111973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 1513.154299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 1513.202554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 1513.250407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_no_route -r 2 [ 1513.306859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_no_route -r 2 [ 1513.379968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_no_route -r 2 [ 1513.419681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_no_route -r 2 [ 1513.466186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1513.753539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1513.786930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1513.812836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1513.843975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1513.871299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1513.900599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1513.929975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1513.954439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1513.980172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1514.017032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1514.049474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1514.095934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1514.125897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1514.161207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1514.567736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 1514.601374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 1514.633608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 1514.676751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 1514.703642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e local_timeout -r 2 [ 1514.766557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e local_timeout -r 2 [ 1514.809885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e local_timeout -r 2 [ 1514.847793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e local_timeout -r 2 [ 1514.877507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1514.891603] LNet: There was an unexpected network error while writing to 192.168.202.135: rc = -22 [ 1514.896371] LNet: Skipped 1 previous similar message [ 1515.121928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1515.144402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1515.191064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1515.233519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1515.273336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1515.306740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1515.335220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1515.367074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1515.392191] LNet: 87521:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1515.395508] LNet: 87521:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1515.408978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1515.438781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1515.475610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1515.508410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1515.535665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1515.575840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1516.010575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 1516.048461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 1516.081658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 1516.108504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 1516.126937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp -m GET -e remote_dropped -r 2 [ 1516.166125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.135@tcp1 -m GET -e remote_dropped -r 2 [ 1516.205706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp -m GET -e remote_dropped -r 2 [ 1516.237475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.35@tcp1 -d 192.168.202.35@tcp1 -m GET -e remote_dropped -r 2 [ 1516.260544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.35@tcp 192.168.202.135@tcp [ 1516.282847] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 1516.481554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1516.504561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1516.535562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1516.561313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1516.588770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1516.620957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1516.642841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1516.671228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1516.694849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1516.724512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1516.764059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1516.803896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1516.832713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1516.869254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1520.969672] LNet: 87715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1520.978120] LNetError: 87715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1520.990073] LNet: Removed LNI 192.168.202.35@tcp [ 1521.002098] LNet: Removed LNI 192.168.202.35@tcp1 [ 1521.385638] Key type .llcrypt unregistered [ 1521.391706] Key type ._llcrypt unregistered [ 1536.205183] Key type ._llcrypt registered [ 1536.207279] Key type .llcrypt registered [ 1537.539794] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:20:16 (1763346016) [ 1537.824762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1537.838933] alg: No test for adler32 (adler32-zlib) [ 1538.734848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.781940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.806895] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1539.719516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1539.756347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.35@tcp [ 1539.775621] LNetError: 88690:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1539.828168] LNet: 1 local NIs in recovery (showing 1): 192.168.202.35@tcp [ 1539.893469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.35@tcp [ 1540.013130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1546.212459] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:20:25 (1763346025) [ 1546.709382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.741483] LNet: 89279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.750226] LNetError: 89279:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.774774] LNet: Removed LNI 192.168.202.35@tcp [ 1546.815721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.853462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.887855] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1547.533492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1547.564210] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1548.486866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.35@tcp [ 1548.506369] LNetError: 89545:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.202.35@tcp to recovery [ 1548.896267] LNet: There was an unexpected network error while writing to 192.168.202.35: rc = -22 [ 1549.704641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.35@tcp [ 1557.864214] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:20:37 (1763346037) [ 1559.555252] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg235-server [ 1561.061382] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:20:40 (1763346040) [ 1561.203609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.262698] LNet: 90500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1561.268250] LNetError: 90500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1561.279580] LNet: Removed LNI 192.168.202.35@tcp [ 1561.297265] LNet: Removed LNI 192.168.202.35@tcp1 [ 1561.318915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.367604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.395519] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1562.074990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1562.132530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1562.182484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1562.252102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1562.304646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1562.351461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1567.778129] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:20:47 (1763346047) [ 1567.913412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.944524] LNet: 91221:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.967686] LNetError: 91221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1567.973640] LNetError: 91221:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1567.992634] LNet: Removed LNI 192.168.202.35@tcp [ 1568.016424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.086526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1568.122333] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1573.967591] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:20:53 (1763346053) [ 1574.142249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.160409] LNet: 91929:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1574.165907] LNetError: 91929:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1574.181576] LNet: Removed LNI 192.168.202.35@tcp [ 1574.205421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.264069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.288193] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1579.890947] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:20:59 (1763346059) [ 1581.527720] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1583.685371] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:21:03 (1763346063) [ 1584.309920] LNet: 92963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1584.318205] LNetError: 92963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.330862] LNet: Removed LNI 192.168.202.35@tcp [ 1584.945224] Key type .llcrypt unregistered [ 1584.947058] Key type ._llcrypt unregistered [ 1591.290859] Key type ._llcrypt registered [ 1591.292740] Key type .llcrypt registered [ 1591.526950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1591.548207] alg: No test for adler32 (adler32-zlib) [ 1592.506457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1592.548581] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1599.748481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1604.685762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.135@tcp [ 1605.028392] LNet: 93376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.036222] LNetError: 93376:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.056669] LNet: Removed LNI 192.168.202.35@tcp [ 1605.435727] Key type .llcrypt unregistered [ 1605.441193] Key type ._llcrypt unregistered [ 1615.256171] Key type ._llcrypt registered [ 1615.257287] Key type .llcrypt registered [ 1616.435123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1617.731310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1619.290480] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:21:38 (1763346098) [ 1619.743925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1619.759874] alg: No test for adler32 (adler32-zlib) [ 1620.663981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1620.740845] LNet: 94460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1621.367627] Key type .llcrypt unregistered [ 1621.371371] Key type ._llcrypt unregistered [ 1626.259540] Key type ._llcrypt registered [ 1626.260851] Key type .llcrypt registered [ 1626.411581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1626.428370] alg: No test for adler32 (adler32-zlib) [ 1627.327874] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1627.330946] LNet: 94711:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1627.368368] LNetError: 94723:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.39@tcp is being used as a gateway but routing feature is not turned on [ 1627.385771] LNetError: 94723:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.39@tcp has gone from up to down [ 1627.777688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1627.808583] LNet: 94769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1627.816204] LNetError: 94769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1627.840830] LNet: Removed LNI 192.168.202.35@tcp [ 1627.865818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1627.928892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1627.951859] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1627.955142] LNet: 94776:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1627.970690] LNet: 94776:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1627.990511] LNetError: 94776:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.202.36@tcp is a gateway. Can not delete it [ 1628.004421] LNetError: 94776:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.202.36@tcp is a router. Can not be deleted [ 1628.011958] LNet: 94776:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1628.016341] LNetError: 94773:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.202.39@tcp is being used as a gateway but routing feature is not turned on [ 1628.039877] LNetError: 94773:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1628.054776] LNetError: 94773:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.202.39@tcp has gone from up to down [ 1628.068199] LNetError: 94773:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1633.810233] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:21:53 (1763346113) [ 1635.796814] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg235-server [ 1637.446712] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:21:57 (1763346117) [ 1639.209984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg235-server [ 1640.792108] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:22:00 (1763346120) [ 1642.284271] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 1644.199386] Key type lgssc registered [ 1645.238849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1661.773663] Key type lgssc unregistered [ 1662.112079] LNet: 97195:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1662.124378] LNetError: 97195:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.141898] LNet: Removed LNI 192.168.202.35@tcp [ 1662.693709] Key type .llcrypt unregistered [ 1662.695144] Key type ._llcrypt unregistered [ 1675.891203] Key type ._llcrypt registered [ 1675.892963] Key type .llcrypt registered [ 1677.095472] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:22:36 (1763346156) [ 1677.606300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.619880] alg: No test for adler32 (adler32-zlib) [ 1678.617496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.712947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1678.800164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1678.854917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1678.947662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1685.545411] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:22:45 (1763346165) [ 1685.902107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.931494] LNet: 98598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.961261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.036973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.078611] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1687.169154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.187711] LNet: 98742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1687.191938] LNetError: 98742:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.229159] LNet: Removed LNI 192.168.202.35@tcp [ 1687.246059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.321050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1687.365922] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1687.869485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.887143] LNetError: 98770:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1687.900514] LNet: Removed LNI 192.168.202.35@tcp [ 1687.928930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.017430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1688.056904] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1695.922902] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:22:55 (1763346175) [ 1696.284698] LNet: 99362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.288262] LNet: 99362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1696.292721] LNetError: 99362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1696.335027] LNet: Removed LNI 192.168.202.35@tcp [ 1697.139594] Key type .llcrypt unregistered [ 1697.141038] Key type ._llcrypt unregistered [ 1703.502932] Key type ._llcrypt registered [ 1703.504522] Key type .llcrypt registered [ 1703.672399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1703.685738] alg: No test for adler32 (adler32-zlib) [ 1704.627193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.707174] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1704.709455] LNet: *** cfs_fail_loc=e003, val=100*** [ 1705.389278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1705.403960] LNet: *** cfs_fail_loc=e003, val=100*** [ 1705.405982] LNet: Skipped 2 previous similar messages [ 1705.415396] Ping reply received with mapping: [ 1705.418521] found metadata reply NI[192.168.202.35@tcp] [ 1705.421210] 23 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1705.424131] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1705.429523] 5a 5a 5a 5a 5a 5a 5a 5a [ 1705.573261] LNet: 99756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.582626] LNetError: 99756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.604136] LNet: Removed LNI 192.168.202.35@tcp [ 1706.175752] Key type .llcrypt unregistered [ 1706.183532] Key type ._llcrypt unregistered [ 1716.881037] Key type ._llcrypt registered [ 1716.883166] Key type .llcrypt registered [ 1718.184746] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:23:17 (1763346197) [ 1718.857436] Key type .llcrypt unregistered [ 1718.859110] Key type ._llcrypt unregistered [ 1724.482033] Key type ._llcrypt registered [ 1724.483718] Key type .llcrypt registered [ 1724.670767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1724.687313] alg: No test for adler32 (adler32-zlib) [ 1725.811582] LNetError: 100791:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1726.197660] Key type .llcrypt unregistered [ 1726.203390] Key type ._llcrypt unregistered [ 1731.062516] Key type ._llcrypt registered [ 1731.064467] Key type .llcrypt registered [ 1732.375304] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:23:31 (1763346211) [ 1732.877316] Key type .llcrypt unregistered [ 1732.878957] Key type ._llcrypt unregistered [ 1737.628675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1737.630611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1740.226605] Key type ._llcrypt registered [ 1740.228187] Key type .llcrypt registered [ 1740.421703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1740.433254] alg: No test for adler32 (adler32-zlib) [ 1741.376842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.431444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1750.433613] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:23:50 (1763346230) [ 1750.780541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1750.800476] LNet: 102563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.810782] LNetError: 102563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1750.830114] LNet: Removed LNI 10.1.2.3@tcp [ 1750.851646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.498823] Key type .llcrypt unregistered [ 1751.500290] Key type ._llcrypt unregistered [ 1756.588790] Key type ._llcrypt registered [ 1756.592093] Key type .llcrypt registered [ 1756.719670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1756.728944] alg: No test for adler32 (adler32-zlib) [ 1757.667943] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1757.728814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1757.909429] LNet: 102848:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.922991] LNetError: 102848:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.950506] LNet: Removed LNI 192.168.202.35@tcp [ 1758.543505] Key type .llcrypt unregistered [ 1758.546452] Key type ._llcrypt unregistered [ 1768.219996] Key type ._llcrypt registered [ 1768.223240] Key type .llcrypt registered [ 1769.413827] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:24:08 (1763346248) [ 1769.991625] Key type .llcrypt unregistered [ 1769.993388] Key type ._llcrypt unregistered [ 1775.442910] Key type ._llcrypt registered [ 1775.444691] Key type .llcrypt registered [ 1775.691987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1775.719346] alg: No test for adler32 (adler32-zlib) [ 1783.025888] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:24:22 (1763346262) [ 1783.297156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.355141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.415440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1783.435167] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1784.048346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.074651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1784.099827] LNet: Removed LNI 192.168.202.35@tcp [ 1784.323698] LNet: 104584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.609489] Key type .llcrypt unregistered [ 1784.611311] Key type ._llcrypt unregistered [ 1794.306103] Key type ._llcrypt registered [ 1794.307748] Key type .llcrypt registered [ 1795.433266] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:24:35 (1763346275) [ 1795.863411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.878698] alg: No test for adler32 (adler32-zlib) [ 1796.802968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.875242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.900372] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1797.512454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1797.536991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.35@tcp [ 1797.553793] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1797.557068] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1797.559270] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1797, 0, 0 [ 1797.563328] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1797.566389] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1797.571163] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1797.573791] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1797, 0, 0 [ 1797.579230] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1797.583102] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1797.589033] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1797.592921] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1797, 0, 1847 [ 1797.596555] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1797.599909] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1797.604041] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1797.607093] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1797, 0, 1847 [ 1797.612105] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1803.511510] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:24:43 (1763346283) [ 1803.824865] LNet: 106126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.830932] LNetError: 106126:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.848884] LNet: Removed LNI 192.168.202.35@tcp [ 1804.256225] Key type .llcrypt unregistered [ 1804.258062] Key type ._llcrypt unregistered [ 1808.472380] Key type ._llcrypt registered [ 1808.473964] Key type .llcrypt registered [ 1808.739294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1808.754517] alg: No test for adler32 (adler32-zlib) [ 1809.642138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1809.667440] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1815.518557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 1822.888456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.927128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1822.957198] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 1823.683093] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1823.749099] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1823.752339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1823.754872] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1826.031288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1826.054402] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1835.645904] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:25:15 (1763346315) [ 1835.735908] LNet: 107620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.749913] LNetError: 107620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.761618] LNet: Removed LNI 192.168.202.35@tcp [ 1836.831437] LNet: Removed LNI 192.168.202.35@tcp1 [ 1836.835887] LNetError: 107620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1836.843965] LNetError: 107620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1836.861044] LNet: Removed LNI 10.1.2.3@tcp99 [ 1837.589231] Key type .llcrypt unregistered [ 1837.591102] Key type ._llcrypt unregistered [ 1842.990198] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1842.997636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1845.630445] Key type ._llcrypt registered [ 1845.639739] Key type .llcrypt registered [ 1845.859411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1845.873523] alg: No test for adler32 (adler32-zlib) [ 1846.863256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.963474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1847.022653] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1847.717293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1847.748865] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1850.600423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1850.664537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1850.710463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1850.750750] LNetError: 108369:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1850.816747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1850.913981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1851.553897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1851.695879] LNet: 108389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1851.709314] LNetError: 108389:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1851.725300] LNet: Removed LNI 192.168.202.35@tcp [ 1851.754265] LNet: Removed LNI 10.1.2.3@tcp [ 1852.953751] Key type .llcrypt unregistered [ 1852.955371] Key type ._llcrypt unregistered [ 1867.360297] Key type ._llcrypt registered [ 1867.362346] Key type .llcrypt registered [ 1868.793572] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:25:48 (1763346348) [ 1869.401714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.423831] alg: No test for adler32 (adler32-zlib) [ 1870.426688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.502857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1870.532647] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1871.224223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.35@tcp [ 1871.294717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg235-client@tcp [ 1877.945401] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:25:57 (1763346357) [ 1878.168111] LNet: 109919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1878.173690] LNetError: 109919:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.188423] LNet: Removed LNI 192.168.202.35@tcp [ 1878.777261] Key type .llcrypt unregistered [ 1878.779406] Key type ._llcrypt unregistered [ 1884.601162] Key type ._llcrypt registered [ 1884.613492] Key type .llcrypt registered [ 1884.848730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.871175] alg: No test for adler32 (adler32-zlib) [ 1885.792656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1885.807743] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1885.829776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1891.675179] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:26:11 (1763346371) [ 1893.050482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1894.464240] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:26:14 (1763346374) [ 1896.029629] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1897.859545] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:26:17 (1763346377) [ 1898.168389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.183824] LNet: 111333:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.187709] LNetError: 111333:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.213297] LNet: Removed LNI 192.168.202.35@tcp [ 1898.229570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1898.294895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1898.316279] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1899.099559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1899.138960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1905.516594] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:26:25 (1763346385) [ 1905.722310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.755699] LNet: 112042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.763290] LNetError: 112042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1905.787980] LNet: Removed LNI 192.168.202.35@tcp [ 1905.808700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.853922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1905.888348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1906.603804] Key type .llcrypt unregistered [ 1906.606540] Key type ._llcrypt unregistered [ 1916.179929] Key type ._llcrypt registered [ 1916.184979] Key type .llcrypt registered [ 1917.266751] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:26:36 (1763346396) [ 1917.564546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1917.579606] alg: No test for adler32 (adler32-zlib) [ 1918.501979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.554614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.576454] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1919.218306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1919.310112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1925.193608] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:26:44 (1763346404) [ 1925.282284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1925.301600] LNet: 113586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1925.307794] LNetError: 113586:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1925.333362] LNet: Removed LNI 192.168.202.35@tcp [ 1925.399663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.442543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1925.465063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1930.917201] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:26:50 (1763346410) [ 1931.247233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1931.268241] LNet: 114173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.292214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1931.362109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1931.747251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1931.782132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1931.837547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1932.169232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1932.202748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1932.253089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.273218] LNet: 114206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.293898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.371587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1932.705513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1932.740570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1932.802802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1933.060903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1933.113344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1938.499280] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:26:58 (1763346418) [ 1938.690846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.711609] LNet: 114806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.725583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.765323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.804700] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1939.493620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1939.519547] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1940.255602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1940.303230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1940.492135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1940.511235] LNetError: 115085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.535433] LNet: Removed LNI 192.168.202.35@tcp [ 1941.537246] LNetError: 115085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.550506] LNet: Removed LNI 192.168.202.35@tcp2 [ 1941.581413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.638946] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1941.645964] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1941.651129] LNet: 115091:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1941.933852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1942.091895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1943.135101] LNet: Removed LNI 192.168.202.35@tcp [ 1943.148187] LNet: Removed LNI 192.168.202.35@tcp2 [ 1943.168253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.241153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1943.279983] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1943.819625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1943.831192] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1944.395441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1944.427452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1944.470020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1944.608194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1944.636605] LNet: 115413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.640111] LNet: 115413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1944.643017] LNetError: 115413:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.645576] LNetError: 115413:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1944.655521] LNet: Removed LNI 192.168.202.35@tcp [ 1945.701667] LNet: Removed LNI 192.168.202.35@tcp2 [ 1945.718347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.765925] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1945.769319] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1945.773174] LNet: 115419:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1946.031872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1946.144342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.158395] LNet: Removed LNI 192.168.202.35@tcp [ 1947.174416] LNet: Removed LNI 192.168.202.35@tcp2 [ 1947.191816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1947.220300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1947.242941] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1947.748647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1947.761320] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1948.426311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1948.474487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1948.648982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.663471] LNetError: 115735:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.667194] LNetError: 115735:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1948.670580] LNet: Removed LNI 192.168.202.35@tcp [ 1948.689769] LNet: Removed LNI 192.168.202.35@tcp2 [ 1948.707289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.758740] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1948.762274] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1948.764429] LNet: 115741:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1949.079842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1949.285867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1950.368017] LNet: Removed LNI 192.168.202.35@tcp [ 1950.385072] LNet: Removed LNI 192.168.202.35@tcp2 [ 1950.416084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1950.452967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1950.482834] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1951.096843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1951.127493] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1951.808708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1951.852398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1951.918196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1952.109741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.183169] LNet: Removed LNI 192.168.202.35@tcp [ 1954.217525] LNet: Removed LNI 192.168.202.35@tcp2 [ 1954.265892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1954.351590] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1954.359545] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1954.363335] LNet: 116069:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1954.843831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1955.004816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.034216] LNet: 116106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.037787] LNet: 116106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1955.045286] LNet: Removed LNI 192.168.202.35@tcp [ 1956.076239] LNet: Removed LNI 192.168.202.35@tcp2 [ 1956.137515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1956.243719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1956.272840] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1957.099161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1957.123080] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1958.090549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1958.117624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1958.160730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1958.562612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.582121] LNetError: 116403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.593651] LNetError: 116403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1958.597282] LNet: Removed LNI 192.168.202.35@tcp [ 1958.609841] LNet: Removed LNI 192.168.202.35@tcp2 [ 1958.626499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.679429] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1958.686715] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1958.691809] LNet: 116409:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1959.185918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1959.474966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1960.543142] LNet: Removed LNI 192.168.202.35@tcp [ 1960.557846] LNet: Removed LNI 192.168.202.35@tcp2 [ 1960.602577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1960.697977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.734075] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1961.462787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1961.504805] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1962.328943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1962.366061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1962.398094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1962.451886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1962.789952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.872799] LNet: Removed LNI 192.168.202.35@tcp [ 1964.903633] LNet: Removed LNI 192.168.202.35@tcp2 [ 1964.959400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1965.041115] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1965.053888] LNet: Added LNI 192.168.202.35@tcp2 [8/256/0/180] [ 1965.686299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1972.891527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1974.574128] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:27:34 (1763346454) [ 1974.775188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1974.793637] LNet: 117527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.801238] LNet: 117527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1974.808723] LNetError: 117527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1974.815732] LNetError: 117527:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1974.825494] LNet: Removed LNI 192.168.202.35@tcp [ 1975.850190] LNet: Removed LNI 192.168.202.35@tcp2 [ 1975.885879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1976.182769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1976.206058] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1976.814236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1976.828297] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1977.512397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.538079] LNet: Removed LNI 192.168.202.35@tcp [ 1977.546380] LNet: Removed LNI 10.1.2.3@tcp [ 1977.568265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.618820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1977.654919] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1977.659976] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1978.065226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1978.083530] LNet: Removed LNI 192.168.202.35@tcp [ 1978.100677] LNet: Removed LNI 10.1.2.3@tcp [ 1983.656119] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:27:43 (1763346463) [ 1983.901660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1983.932080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1984.011114] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1984.139446] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1984.141475] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1984.143883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1986.473818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1986.520502] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 1987.119403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1987.138895] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1988.928380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.35@tcp [ 1988.967104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1989.028465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.35@tcp -m PUT -e local_timeout -r 1 [ 1989.072694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1989.125052] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1989.127246] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1989.379616] LNetError: 118461:0:(lib-move.c:4947:lnet_parse()) 192.168.202.35@tcp, src 192.168.202.35@tcp: Dropping PUT (error -108 looking up sender) [ 1991.455239] LNet: Removed LNI 192.168.202.35@tcp [ 1991.465165] LNet: Removed LNI 10.1.2.3@tcp [ 1991.745155] Key type .llcrypt unregistered [ 1991.746971] Key type ._llcrypt unregistered [ 2001.492752] Key type ._llcrypt registered [ 2001.499377] Key type .llcrypt registered [ 2002.877952] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:28:02 (1763346482) [ 2003.292424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2003.308173] alg: No test for adler32 (adler32-zlib) [ 2004.224390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.545939] LNet: 119768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.039474] Key type .llcrypt unregistered [ 2005.042558] Key type ._llcrypt unregistered [ 2009.819590] Key type ._llcrypt registered [ 2009.831675] Key type .llcrypt registered [ 2010.022433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2010.036572] alg: No test for adler32 (adler32-zlib) [ 2010.905310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2010.925291] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2017.645605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2017.823644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2018.281828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2018.501185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2018.535389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2018.567191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2018.592132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2018.615688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2018.648285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2018.672424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2018.719495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2018.744981] LNet: 120213:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2018.758414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2018.785168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2018.808146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2018.830749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2018.858095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2018.886570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2019.406146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2019.573048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2019.598757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2019.624422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2019.648846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2019.680692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2019.709433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2019.733251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2019.762754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2019.790284] LNet: 120330:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2019.816257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2019.858216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2019.903711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2019.931157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2019.964753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2020.002117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2020.504639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2020.719164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2020.750440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2020.774630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2020.802879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2020.832453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2020.865889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2020.895239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2020.922848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2020.935391] LNet: 120448:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2020.955387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2020.990277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2021.024892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2021.054779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2021.086557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2021.110763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2021.554863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2021.789393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2021.815461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2021.843125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2021.903490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2021.939383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2021.989685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2022.020817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2022.052594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2022.085434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2022.123124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2022.145584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2022.175875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2022.187088] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2022.202684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2022.242996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2022.520134] LNet: 120610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.531199] LNetError: 120610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.547785] LNet: Removed LNI 192.168.202.35@tcp [ 2022.880454] Key type .llcrypt unregistered [ 2022.882813] Key type ._llcrypt unregistered [ 2037.239897] Key type ._llcrypt registered [ 2037.241246] Key type .llcrypt registered [ 2038.632281] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:28:37 (1763346517) [ 2039.150596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.171725] alg: No test for adler32 (adler32-zlib) [ 2040.108704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.534176] LNet: 121493:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.323083] Key type .llcrypt unregistered [ 2041.325147] Key type ._llcrypt unregistered [ 2046.616463] Key type ._llcrypt registered [ 2046.623325] Key type .llcrypt registered [ 2046.789624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2046.802044] alg: No test for adler32 (adler32-zlib) [ 2047.741176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2047.768944] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2054.598343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.135@tcp [ 2062.490820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2062.537808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2062.560481] LNet: Added LNI 192.168.202.35@tcp1 [8/256/0/180] [ 2063.466648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2064.052924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.135@tcp [ 2064.159605] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.135@tcp [ 2064.298271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2064.334402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2064.374722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2064.404754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2064.429284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2064.457198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2064.485360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2064.513919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2064.527835] LNet: 122140:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2064.540958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2064.567552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2064.608602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2064.640879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2064.714282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2064.739610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2065.316727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2065.548466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2065.572796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2065.600269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2065.626671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2065.673737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2065.721788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2065.769689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2065.836341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2065.862791] LNet: 122257:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2065.893614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2065.933963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2065.978830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2066.038241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2066.110137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2066.152968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2066.850245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.135@tcp [ 2067.157594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2067.203973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2067.242624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2067.270677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2067.298882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2067.326885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2067.359181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2067.389711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2067.399785] LNet: 122374:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2067.423436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2067.451356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2067.491817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2067.519486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2067.536567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2067.574848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2068.127471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2068.355632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2068.383534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2068.424092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2068.440243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2068.461092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2068.508263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2068.546741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2068.583331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2068.613835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2068.650299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2068.686164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2068.731291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2068.773645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2068.806352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2072.955669] LNet: 122571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.966377] LNetError: 122571:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2072.973097] LNet: Removed LNI 192.168.202.35@tcp [ 2072.981820] LNet: Removed LNI 192.168.202.35@tcp1 [ 2073.386161] Key type .llcrypt unregistered [ 2073.387827] Key type ._llcrypt unregistered [ 2086.893328] Key type ._llcrypt registered [ 2086.894613] Key type .llcrypt registered [ 2088.129483] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1897 sec ========== 21:29:27 (1763346567) [ 2088.994455] Key type .llcrypt unregistered [ 2088.996890] Key type ._llcrypt unregistered [ 2099.376507] Key type ._llcrypt registered [ 2099.377995] Key type .llcrypt registered [ 2099.628308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2099.642182] alg: No test for adler32 (adler32-zlib) [ 2100.706636] Lustre: Lustre: Build Version: 2.16.61_46_gcd0e473 [ 2100.917458] LNet: Added LNI 192.168.202.35@tcp [8/256/0/180] [ 2102.599756] Key type lgssc registered [ 2103.686934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2160.535433] Lustre: Mounted lustre-client [ 2164.916777] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2177.274133] LustreError: 125803:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8ce0cb486000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2177.284660] LustreError: 125803:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2177.328506] Lustre: Unmounted lustre-client [ 2216.589331] Key type lgssc unregistered [ 2216.975903] LNet: 126286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2216.995571] LNetError: 126286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.039454] LNet: Removed LNI 192.168.202.35@tcp [ 2217.929139] Key type .llcrypt unregistered [ 2217.934777] Key type ._llcrypt unregistered