[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 434865852 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002411] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008014] pid_max: default: 32768 minimum: 301 [ 0.010041] LSM: Security Framework initializing [ 0.011048] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013079] *** VALIDATE selinux *** [ 0.021467] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026508] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028162] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029124] *** VALIDATE tmpfs *** [ 0.030419] *** VALIDATE proc *** [ 0.032129] *** VALIDATE cgroup *** [ 0.032955] *** VALIDATE cgroup2 *** [ 0.033280] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035051] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037024] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.040907] debug: unmapping init [mem 0xffffffff90c59000-0xffffffff90c60fff] [ 0.042801] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043649] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045016] ... bit width: 48 [ 0.046020] ... generic registers: 4 [ 0.047015] ... value mask: 0000ffffffffffff [ 0.048015] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050010] ... event mask: 000000070000000f [ 0.051288] rcu: Hierarchical SRCU implementation. [ 0.053475] smp: Bringing up secondary CPUs ... [ 0.054514] x86: Booting SMP configuration: [ 0.055037] .... node #0, CPUs: #1 #2 #3 [ 0.058614] smp: Brought up 1 node, 4 CPUs [ 0.060018] smpboot: Max logical packages: 1 [ 0.061020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.222128] node 0 deferred pages initialised in 159ms [ 0.226010] devtmpfs: initialized [ 0.227316] x86/mm: Memory block size: 128MB [ 0.229234] gcov: version magic: 0x41383552 [ 0.231412] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.232088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.233228] pinctrl core: initialized pinctrl subsystem [ 0.235293] [ 0.235939] ************************************************************* [ 0.238023] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.241016] ** ** [ 0.242012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.244021] ** ** [ 0.246014] ** This means that this kernel is built to expose internal ** [ 0.248014] ** IOMMU data structures, which may compromise security on ** [ 0.251021] ** your system. ** [ 0.253013] ** ** [ 0.256013] ** If you see this message and you are not debugging the ** [ 0.258011] ** kernel, report this immediately to your vendor! ** [ 0.260015] ** ** [ 0.262013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.264013] ************************************************************* [ 0.265623] NET: Registered protocol family 16 [ 0.267285] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.269041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.271062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.274450] cpuidle: using governor menu [ 0.276838] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.280514] PCI: Using configuration type 1 for base access [ 0.282172] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.291226] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.294043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.299122] cryptd: max_cpu_qlen set to 1000 [ 0.302266] ACPI: Added _OSI(Module Device) [ 0.304021] ACPI: Added _OSI(Processor Device) [ 0.306016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.308020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.313541] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.319491] ACPI: Interpreter enabled [ 0.321085] ACPI: PM: (supports S0 S3 S4 S5) [ 0.323022] ACPI: Using IOAPIC for interrupt routing [ 0.325124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.330455] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.339413] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.342063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.345033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.348105] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.352585] acpiphp: Slot [2] registered [ 0.354156] acpiphp: Slot [5] registered [ 0.356171] acpiphp: Slot [6] registered [ 0.357130] acpiphp: Slot [3] registered [ 0.359167] acpiphp: Slot [4] registered [ 0.360160] acpiphp: Slot [7] registered [ 0.362232] acpiphp: Slot [8] registered [ 0.364146] acpiphp: Slot [9] registered [ 0.366152] acpiphp: Slot [10] registered [ 0.368183] acpiphp: Slot [11] registered [ 0.369163] acpiphp: Slot [12] registered [ 0.371189] acpiphp: Slot [13] registered [ 0.373151] acpiphp: Slot [14] registered [ 0.375223] acpiphp: Slot [15] registered [ 0.377193] acpiphp: Slot [16] registered [ 0.379156] acpiphp: Slot [17] registered [ 0.380131] acpiphp: Slot [18] registered [ 0.382230] acpiphp: Slot [19] registered [ 0.384138] acpiphp: Slot [20] registered [ 0.386262] acpiphp: Slot [21] registered [ 0.388128] acpiphp: Slot [22] registered [ 0.389118] acpiphp: Slot [23] registered [ 0.391145] acpiphp: Slot [24] registered [ 0.393142] acpiphp: Slot [25] registered [ 0.394211] acpiphp: Slot [26] registered [ 0.395070] acpiphp: Slot [27] registered [ 0.397176] acpiphp: Slot [28] registered [ 0.398128] acpiphp: Slot [29] registered [ 0.400122] acpiphp: Slot [30] registered [ 0.401126] acpiphp: Slot [31] registered [ 0.402079] PCI host bridge to bus 0000:00 [ 0.404028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.406029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.409026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.411027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.414034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.417030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.418166] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.421100] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.424000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.431015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.434944] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.438020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.441028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.444021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.448384] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.450835] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.454052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.458084] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.463023] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.474023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.479018] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.485413] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.513018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.519018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.534968] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.545802] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.552022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.558022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.579018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.590142] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.592404] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.595451] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.598408] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.601307] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.606128] iommu: Default domain type: Passthrough [ 0.609471] SCSI subsystem initialized [ 0.612222] ACPI: bus type USB registered [ 0.614100] usbcore: registered new interface driver usbfs [ 0.616167] usbcore: registered new interface driver hub [ 0.618191] usbcore: registered new device driver usb [ 0.620268] pps_core: LinuxPPS API ver. 1 registered [ 0.623015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.626067] PTP clock support registered [ 0.628151] EDAC MC: Ver: 3.0.0 [ 0.630444] PCI: Using ACPI for IRQ routing [ 0.632962] NetLabel: Initializing [ 0.635030] NetLabel: domain hash size = 128 [ 0.636019] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.638122] NetLabel: unlabeled traffic allowed by default [ 0.642073] vgaarb: loaded [ 0.644405] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.646020] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.653773] clocksource: Switched to clocksource kvm-clock [ 0.768227] VFS: Disk quotas dquot_6.6.0 [ 0.770094] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.772564] *** VALIDATE ramfs *** [ 0.773948] *** VALIDATE hugetlbfs *** [ 0.776365] pnp: PnP ACPI init [ 0.779129] pnp: PnP ACPI: found 6 devices [ 0.800592] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.803831] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.806159] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.808283] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.810638] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.812878] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.815614] NET: Registered protocol family 2 [ 0.818046] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.822753] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.826234] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.831571] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.834971] TCP: Hash tables configured (established 65536 bind 65536) [ 0.837967] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.841416] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.844450] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.848406] NET: Registered protocol family 1 [ 0.852230] RPC: Registered named UNIX socket transport module. [ 0.854918] RPC: Registered udp transport module. [ 0.856924] RPC: Registered tcp transport module. [ 0.858827] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.861641] NET: Registered protocol family 44 [ 0.863424] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.865586] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.867636] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.870134] PCI: CLS 0 bytes, default 64 [ 0.871861] Unpacking initramfs... [ 2.388682] debug: unmapping init [mem 0xffff92e97cc64000-0xffff92e97ffcffff] [ 2.392677] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.395451] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.399480] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.891323] Initialise system trusted keyrings [ 2.893704] Key type blacklist registered [ 2.897131] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.907893] zbud: loaded [ 2.912241] *** VALIDATE nfs *** [ 2.913679] *** VALIDATE nfs4 *** [ 2.915452] pstore: using deflate compression [ 2.920359] Platform Keyring initialized [ 3.048695] NET: Registered protocol family 38 [ 3.051316] Key type asymmetric registered [ 3.052853] Asymmetric key parser 'x509' registered [ 3.054390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.057912] io scheduler mq-deadline registered [ 3.060182] io scheduler kyber registered [ 3.062359] io scheduler bfq registered [ 3.065163] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.068665] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.071918] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.075724] ACPI: Power Button [PWRF] [ 3.082652] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.089831] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.098795] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.126971] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.155649] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.163027] Non-volatile memory driver v1.3 [ 3.164988] Linux agpgart interface v0.103 [ 3.200400] virtio_blk virtio1: [vda] 136584 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.206074] vda: detected capacity change from 0 to 69931008 [ 3.221338] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.224493] vdb: detected capacity change from 0 to 1073741824 [ 3.236349] libphy: Fixed MDIO Bus: probed [ 3.241381] usbcore: registered new interface driver usbserial_generic [ 3.243748] usbserial: USB Serial support registered for generic [ 3.245759] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.249621] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.251262] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.254035] mousedev: PS/2 mouse device common for all mice [ 3.257470] rtc_cmos 00:05: RTC can wake from S4 [ 3.260402] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.264823] rtc_cmos 00:05: registered as rtc0 [ 3.267529] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.270375] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.271148] intel_pstate: CPU model not supported [ 3.278026] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.281395] hid: raw HID events driver (C) Jiri Kosina [ 3.283917] usbcore: registered new interface driver usbhid [ 3.285348] usbhid: USB HID core driver [ 3.286485] drop_monitor: Initializing network drop monitor service [ 3.288670] Initializing XFRM netlink socket [ 3.290838] NET: Registered protocol family 10 [ 3.293851] Segment Routing with IPv6 [ 3.295340] NET: Registered protocol family 17 [ 3.297928] mpls_gso: MPLS GSO support [ 3.303555] RAS: Correctable Errors collector initialized. [ 3.305595] AVX version of gcm_enc/dec engaged. [ 3.307348] AES CTR mode by8 optimization enabled [ 3.386854] sched_clock: Marking stable (3386816811, 0)->(4328122041, -941305230) [ 3.391205] registered taskstats version 1 [ 3.393389] Loading compiled-in X.509 certificates [ 3.395468] zswap: loaded using pool lzo/zbud [ 3.422085] Key type big_key registered [ 3.434793] Key type encrypted registered [ 3.436517] ima: No TPM chip found, activating TPM-bypass! [ 3.438666] ima: Allocated hash algorithm: sha1 [ 3.440493] ima: No architecture policies found [ 3.442504] evm: Initialising EVM extended attributes: [ 3.444718] evm: security.selinux [ 3.446014] evm: security.ima [ 3.447179] evm: security.capability [ 3.448509] evm: HMAC attrs: 0x1 [ 3.452566] rtc_cmos 00:05: setting system clock to 2026-06-12 03:55:43 UTC (1781236543) [ 3.459641] debug: unmapping init [mem 0xffffffff91c03000-0xffffffff91dfffff] [ 3.463144] debug: unmapping init [mem 0xffffffff90982000-0xffffffff90c58fff] [ 3.472118] Write protecting the kernel read-only data: 28672k [ 3.475980] debug: unmapping init [mem 0xffffffff8f003000-0xffffffff8f1fffff] [ 3.479097] debug: unmapping init [mem 0xffffffff8f914000-0xffffffff8f9fffff] [ 3.510790] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.519106] systemd[1]: Detected virtualization kvm. [ 3.520970] systemd[1]: Detected architecture x86-64. [ 3.522903] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.548748] systemd[1]: No hostname configured. [ 3.550628] systemd[1]: Set hostname to . [ 3.552778] random: systemd: uninitialized urandom read (16 bytes read) [ 3.555356] systemd[1]: Initializing machine ID from random generator. [ 3.604398] random: ln: uninitialized urandom read (6 bytes read) [ 3.697351] random: systemd: uninitialized urandom read (16 bytes read) [ 3.700100] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.706099] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.711758] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.348281] device-mapper: uevent: version 1.0.3 [ 4.350386] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [[ 4.981981] random: fast init done  OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.032445] virtio_net virtio0 ens2: renamed from eth0 [ 5.035058] scsi host0: ata_piix [ 5.038294] scsi host1: ata_piix [ 5.040410] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.044899] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.730926] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.957846] random: crng init done [ 9.959388] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.290721] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.410193] printk: systemd: 22 output lines suppressed due to ratelimiting [ 11.687657] SELinux: Disabled at runtime. [ 11.745938] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.754787] systemd[1]: Detected virtualization kvm. [ 11.756993] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.265088] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.270755] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.276712] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.281172] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.284875] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.292475] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.308234] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Remount Root and Kernel File Systems... [ 12.447692] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Initrd File Systems. Starting Apply Kernel Variables... [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.762368] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.102659] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.138527] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.167219] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.198705] EDAC sbridge: Ver: 1.1.2 [ 14.367221] Key type dns_resolver registered [ 14.666570] NFS: Registering the id_resolver key type [ 14.668804] Key type id_resolver registered [ 14.670527] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg310-client login: [ 72.931292] libcfs: loading out-of-tree module taints kernel. [ 73.273709] Key type ._llcrypt registered [ 73.277311] Key type .llcrypt registered [ 73.978049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 74.014455] alg: No test for adler32 (adler32-zlib) [ 75.660281] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 76.746107] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 78.560155] Key type lgssc registered [ 80.782663] Lustre: Echo OBD driver; http://www.lustre.org/ [ 152.377008] hrtimer: interrupt took 4601212 ns [ 206.061990] Lustre: Mounted lustre-client [ 211.657517] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.294970] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing check_logdir /tmp/testlogs/ [ 231.904228] Lustre: lustre-OST0000-osc-ffff92e9d89f1800: disconnect after 23s idle [ 232.261438] Lustre: DEBUG MARKER: oleg310-client.virtnet: executing yml_node [ 238.197721] Lustre: DEBUG MARKER: Client: 2.17.53.79 [ 242.049192] Lustre: DEBUG MARKER: MDS: 2.17.53.79 [ 245.166661] Lustre: DEBUG MARKER: OSS: 2.17.53.79 [ 247.702704] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Jun 11 23:59:45 EDT 2026 [ 267.378373] Lustre: Unmounted lustre-client [ 288.604325] Key type lgssc unregistered [ 289.000085] LNet: 8378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.011492] LNetError: 8378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 289.032406] LNet: Removed LNI 192.168.203.10@tcp [ 290.091265] Key type .llcrypt unregistered [ 290.095557] Key type ._llcrypt unregistered [ 317.755414] Key type ._llcrypt registered [ 317.757590] Key type .llcrypt registered [ 318.024523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 318.056514] alg: No test for adler32 (adler32-zlib) [ 319.043731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 319.193367] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 319.250456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 324.343802] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 324.507814] LNet: 9270:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.522911] LNetError: 9270:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.542303] LNet: Removed LNI 192.168.203.10@tcp [ 325.359418] Key type .llcrypt unregistered [ 325.367042] Key type ._llcrypt unregistered [ 332.582354] Key type ._llcrypt registered [ 332.590290] Key type .llcrypt registered [ 334.248565] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:01:12 (1781236872) [ 334.600748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 334.624532] alg: No test for adler32 (adler32-zlib) [ 335.583917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.719992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.743344] LNet: 9688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.762537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.866489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 335.899239] LNet: 9695:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 343.202757] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:01:21 (1781236881) [ 343.312369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.348578] LNet: 10267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 343.397338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.489313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 352.816474] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:01:31 (1781236891) [ 353.004750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.043894] LNet: 10860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.102694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.181681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 361.950499] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:01:40 (1781236900) [ 362.020581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.043048] LNet: 11442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 362.066158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 362.165718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 371.428645] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:01:49 (1781236909) [ 371.891720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.924950] LNet: 12031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.989276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.148876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 373.248774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 383.632944] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:02:01 (1781236921) [ 383.858995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.919463] LNet: 12623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.961793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.032529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.439704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.891808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 393.047640] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:02:11 (1781236931) [ 393.188859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.250582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.406387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 401.502540] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:02:19 (1781236939) [ 401.632317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.671558] LNet: 13798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.682902] LNet: 13798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 401.725568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.861128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 401.915359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.476601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 402.534111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 403.058436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 403.141762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.693619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 403.737659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 404.383723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 404.458776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 404.896247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 404.943801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.438590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 405.477247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.196668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 406.227138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 406.744869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 406.827789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 407.516512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 407.556788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 408.262196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 408.331840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 408.968555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 409.027294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 409.462763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 409.533078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 417.416555] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:02:35 (1781236955) [ 417.579595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.674808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.826104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 417.900298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 425.615656] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:02:44 (1781236964) [ 425.726608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.774556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.955076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.014664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 434.149662] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:02:52 (1781236972) [ 434.252714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.275731] LNet: 15654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.289897] LNet: 15654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 434.333251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.461547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 434.513529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 441.892586] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:03:00 (1781236980) [ 442.072351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 442.132878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 442.240932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 442.294081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 449.875732] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:03:08 (1781236988) [ 450.151780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.213930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.303954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 450.338740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 457.282132] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:03:16 (1781236996) [ 457.464606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.527451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.585975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 457.620903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 464.202186] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:03:23 (1781237003) [ 464.298797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.345894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.460823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 464.973061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 472.353436] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:03:31 (1781237011) [ 472.550565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.622559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.881237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.183481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 480.091467] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:03:38 (1781237018) [ 480.185853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 480.241700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 480.427223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 480.713198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 488.025406] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:03:46 (1781237026) [ 488.229701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 488.258729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 488.697709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 488.998316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 496.236777] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:03:54 (1781237034) [ 496.420666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.483891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.569592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 496.616893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 503.663153] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:04:02 (1781237042) [ 503.778756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.810565] LNet: 21077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.818437] LNet: 21077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 503.848702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.977677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 504.035685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 511.120616] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:04:10 (1781237050) [ 511.507545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.569476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.634668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 518.001158] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:04:16 (1781237056) [ 518.363464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.419688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.526468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 525.206683] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:04:23 (1781237063) [ 525.561933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.636522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.745949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 526.327479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 532.883502] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:04:31 (1781237071) [ 533.007615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.067709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.167863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 533.216323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 539.520604] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:04:38 (1781237078) [ 539.931266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.961987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.059276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 540.092379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 547.056829] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:04:45 (1781237085) [ 547.121797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.156263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.229678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 547.267884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 554.623474] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:04:52 (1781237092) [ 554.730530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.772665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.920337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 554.983597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 562.952744] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:05:01 (1781237101) [ 563.203570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.264880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.361331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 563.441375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 563.460425] LNetError: 25808:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 563.500849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 563.552709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 569.429062] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:05:08 (1781237108) [ 569.735346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.789650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.896261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aQthG3kxV6 [ 569.982612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aQthG3kxV6 [ 570.042612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.aQthG3kxV6 [ 577.168544] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:05:15 (1781237115) [ 577.412826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 577.468710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 577.567419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 577.672452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 577.777106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 584.001323] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:05:22 (1781237122) [ 584.198663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.225870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.295570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 587.724513] LNet: Message forwarding will be enabled in 2s [ 590.818088] LNet: Message forwarding enabled [ 591.304801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.324898] LNet: Message forwarding will stop in 2s to 6s [ 595.922193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.583110] LNet: Message forwarding disabled [ 598.773690] LNet: Message forwarding will be enabled in 2s [ 602.721165] LNet: Message forwarding enabled [ 605.756380] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:05:44 (1781237144) [ 605.886090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.350953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.424818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 613.696679] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:05:52 (1781237152) [ 613.876442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 613.941061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.029455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 614.061972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 614.085438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 614.121598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 614.182904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 614.235049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 614.277933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 614.331290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 614.368113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 614.412285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 614.441840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 614.476904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 614.521952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 614.573132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 614.619641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 614.710722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 614.765287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 614.804151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 614.838619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 614.874209] LNetError: 28879:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 621.012782] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:05:59 (1781237159) [ 621.108806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.205253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.303367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 628.725935] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:06:07 (1781237167) [ 630.604229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg310-server [ 632.475947] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:06:11 (1781237171) [ 632.589089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.611757] LNet: 30376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.617251] LNet: 30376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 14 previous similar messages [ 632.639754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.727829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 632.754841] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 634.831071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[11-15]@tcp [ 634.860177] LNet: 30596:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 634.910650] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.14@tcp is being used as a gateway but routing feature is not turned on [ 634.926482] LNetError: 30385:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.14@tcp has gone from up to down [ 635.939361] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.13@tcp is being used as a gateway but routing feature is not turned on [ 635.961852] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 636.981740] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 637.004584] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 639.019899] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.13@tcp is being used as a gateway but routing feature is not turned on [ 639.037560] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 641.685837] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:06:20 (1781237180) [ 643.114633] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.203.15@tcp is being used as a gateway but routing feature is not turned on [ 643.137149] LNetError: 30385:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 643.602986] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg310-server [ 645.290682] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:06:23 (1781237183) [ 645.397612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.427614] LNetError: 31524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.452392] LNet: Removed LNI 192.168.203.10@tcp [ 645.516781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.616506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 645.655597] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 646.259266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[11-16/2]@tcp [ 646.284896] LNet: 31648:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 646.299897] LNet: 31648:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 646.321994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[11-16/2]@tcp [ 646.326715] LNetError: 31535:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.13@tcp has gone from up to down [ 646.371721] LNetError: 31535:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 653.082368] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:06:31 (1781237191) [ 653.362992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.391847] LNetError: 32241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 653.416561] LNet: Removed LNI 192.168.203.10@tcp [ 653.451462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.543333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 653.599673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.659768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 653.772174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 653.868352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 653.946899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.019216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 654.136134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.198766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.283167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.346454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.385701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.460415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 654.583356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.632822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.745896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.865485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 654.919581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.970602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 655.063392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.121693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.217462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.339486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.392657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.459880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 655.540150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.596374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.665479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 655.777452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.817682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.893961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 655.978689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.018840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.079866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.066945] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:06:41 (1781237201) [ 665.211674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg310-server [ 667.138954] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:06:45 (1781237205) [ 669.419182] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg310-server [ 671.271456] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:06:50 (1781237210) [ 672.590978] Key type .llcrypt unregistered [ 672.599162] Key type ._llcrypt unregistered [ 679.022354] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 679.300830] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 679.315272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 679.321956] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.945402] Key type ._llcrypt registered [ 680.950208] Key type .llcrypt registered [ 681.238117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 681.252818] alg: No test for adler32 (adler32-zlib) [ 682.220508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.286250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.332286] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 683.084396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 683.114502] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 685.367270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 685.402958] LNet: Removed LNI 10.1.2.3@tcp [ 685.557651] LNet: 34356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.569598] LNetError: 34356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.589951] LNet: Removed LNI 192.168.203.10@tcp [ 686.407139] Key type .llcrypt unregistered [ 686.409258] Key type ._llcrypt unregistered [ 696.979981] Key type ._llcrypt registered [ 696.981447] Key type .llcrypt registered [ 698.212596] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:07:16 (1781237236) [ 699.014311] Key type .llcrypt unregistered [ 699.021193] Key type ._llcrypt unregistered [ 704.609624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 704.695995] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 704.703869] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 704.710569] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 706.254366] Key type ._llcrypt registered [ 706.260256] Key type .llcrypt registered [ 706.424937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 706.438866] alg: No test for adler32 (adler32-zlib) [ 707.365852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.444362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.470576] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 708.418627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 708.450537] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 711.493636] LNet: 35844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 711.499885] LNetError: 35844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 711.505864] LNet: Removed LNI 192.168.203.10@tcp [ 711.514252] LNet: Removed LNI 10.1.2.3@tcp [ 712.107642] Key type .llcrypt unregistered [ 712.115993] Key type ._llcrypt unregistered [ 721.991675] Key type ._llcrypt registered [ 721.995226] Key type .llcrypt registered [ 723.402951] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:07:41 (1781237261) [ 723.996433] Key type .llcrypt unregistered [ 724.000168] Key type ._llcrypt unregistered [ 729.885098] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 730.047897] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 730.051398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 730.055108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 732.592544] Key type ._llcrypt registered [ 732.596327] Key type .llcrypt registered [ 732.797605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 732.814654] alg: No test for adler32 (adler32-zlib) [ 733.740644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.818297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 733.867921] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 735.897996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 735.918801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 737.931382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 737.950960] LNet: Removed LNI 10.1.2.3@tcp [ 737.961889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 738.024428] LNet: Removed LNI 10.1.2.31@tcp [ 738.039562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 738.072992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 738.854436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 738.877324] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 740.020422] LNet: 37766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.025967] LNetError: 37766:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 740.031349] LNet: Removed LNI 10.1.2.3@tcp [ 740.045737] LNet: Removed LNI 10.1.2.31@tcp [ 740.853970] Key type .llcrypt unregistered [ 740.855911] Key type ._llcrypt unregistered [ 753.837781] Key type ._llcrypt registered [ 753.846446] Key type .llcrypt registered [ 755.344637] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:08:13 (1781237293) [ 755.894484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 755.914499] alg: No test for adler32 (adler32-zlib) [ 756.827617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.878630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.901114] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 757.781849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.814800] LNet: 38719:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.825184] LNetError: 38719:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.837263] LNet: Removed LNI 192.168.203.10@tcp [ 757.854958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.934581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.10@tcp [ 757.979830] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 758.649938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.10@tcp [ 758.698125] LNet: Removed LNI 192.168.203.10@tcp [ 758.778397] LNet: 38854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.450382] Key type .llcrypt unregistered [ 759.453168] Key type ._llcrypt unregistered [ 769.748643] Key type ._llcrypt registered [ 769.750814] Key type .llcrypt registered [ 770.983842] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:08:29 (1781237309) [ 773.108980] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg310-server [ 774.612370] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:08:33 (1781237313) [ 775.185913] Key type .llcrypt unregistered [ 775.189614] Key type ._llcrypt unregistered [ 781.283250] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 781.409454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 781.413318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 781.416666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 784.062684] Key type ._llcrypt registered [ 784.064836] Key type .llcrypt registered [ 784.315478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.330675] alg: No test for adler32 (adler32-zlib) [ 785.299562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.389108] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 786.110332] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 788.135658] LNet: 40692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.141374] LNetError: 40692:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.149104] LNet: Removed LNI 192.168.203.10@tcp1 [ 788.159988] LNet: Removed LNI 10.1.2.3@tcp2 [ 788.705780] Key type .llcrypt unregistered [ 788.709634] Key type ._llcrypt unregistered [ 800.187869] Key type ._llcrypt registered [ 800.194299] Key type .llcrypt registered [ 801.463236] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:09:00 (1781237340) [ 801.961618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 801.976693] alg: No test for adler32 (adler32-zlib) [ 802.932281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.975262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 802.996054] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 803.818093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.840446] LNet: 41660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.848625] LNetError: 41660:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.858870] LNet: Removed LNI 192.168.203.10@tcp [ 803.879395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.963868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 804.012777] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 810.696842] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:09:09 (1781237349) [ 810.783460] LNet: 42243:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.792582] LNetError: 42243:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 810.819669] LNet: Removed LNI 192.168.203.10@tcp [ 811.454486] Key type .llcrypt unregistered [ 811.457076] Key type ._llcrypt unregistered [ 816.794443] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 816.900082] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 816.913377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 816.922156] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 819.603918] Key type ._llcrypt registered [ 819.608463] Key type .llcrypt registered [ 819.842320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 819.857050] alg: No test for adler32 (adler32-zlib) [ 820.802860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.877710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 820.915409] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 821.691031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 821.714655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.048257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.073406] LNet: 42994:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.079937] LNetError: 42994:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.086987] LNet: Removed LNI 192.168.203.10@tcp [ 824.098361] LNet: Removed LNI 10.1.2.3@tcp [ 824.130437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.229793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 824.268576] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 824.278471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 824.894629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 824.909433] LNet: 43021:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.915798] LNetError: 43021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 824.924120] LNetError: 43021:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 824.932069] LNet: Removed LNI 192.168.203.10@tcp [ 824.941285] LNet: Removed LNI 10.1.2.3@tcp [ 824.960429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.027890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 825.059657] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 825.062926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 825.600171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 825.629179] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 826.534062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.563861] LNet: 43184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.572772] LNetError: 43184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.582482] LNetError: 43184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 826.593777] LNet: Removed LNI 192.168.203.10@tcp [ 826.598151] LNet: Removed LNI 10.1.2.3@tcp [ 826.617568] LNet: Removed LNI 192.168.203.10@tcp2 [ 826.633884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.680983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 826.701561] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 826.704238] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.709695] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 834.971384] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:09:33 (1781237373) [ 835.142333] LNet: 43780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 835.166829] LNetError: 43780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 835.176768] LNetError: 43780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 835.186534] LNet: Removed LNI 192.168.203.10@tcp [ 835.189267] LNet: Removed LNI 10.1.2.3@tcp [ 835.203426] LNet: Removed LNI 192.168.203.10@tcp2 [ 835.900676] Key type .llcrypt unregistered [ 835.907816] Key type ._llcrypt unregistered [ 841.754811] Key type ._llcrypt registered [ 841.757146] Key type .llcrypt registered [ 841.927360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.936879] alg: No test for adler32 (adler32-zlib) [ 843.054766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.173257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.219340] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 844.276234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.331836] LNet: 44175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.343224] LNetError: 44175:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.364396] LNet: Removed LNI 192.168.203.10@tcp [ 844.422729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.514620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 844.563363] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 845.401392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.441807] LNet: 44205:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.453121] LNetError: 44205:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.463240] LNet: Removed LNI 192.168.203.10@tcp [ 845.505350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.615706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 845.669953] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 846.325333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 846.379658] LNet: Removed LNI 192.168.203.10@tcp [ 846.405288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.545679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 846.559918] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 847.105138] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 847.289448] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 847.302082] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 847.304408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 849.803820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 849.831413] LNet: 44314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.839153] LNet: 44314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 849.848947] LNetError: 44314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 849.851884] LNetError: 44314:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 849.865859] LNet: Removed LNI 192.168.203.10@tcp [ 849.890467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.969086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 850.006412] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 857.123464] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:09:55 (1781237395) [ 857.292415] LNet: 44910:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.310621] LNetError: 44910:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.338243] LNet: Removed LNI 192.168.203.10@tcp [ 858.191864] Key type .llcrypt unregistered [ 858.193580] Key type ._llcrypt unregistered [ 863.927031] Key type ._llcrypt registered [ 863.930265] Key type .llcrypt registered [ 864.232650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 864.251540] alg: No test for adler32 (adler32-zlib) [ 865.189738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.269272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.306154] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 866.231824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.254156] LNet: 45310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.259633] LNetError: 45310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.281693] LNet: Removed LNI 192.168.203.10@tcp [ 866.301610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.351904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 866.378276] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 872.422478] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:10:11 (1781237411) [ 872.533330] LNet: 45899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.540277] LNetError: 45899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 872.563839] LNet: Removed LNI 192.168.203.10@tcp [ 873.063599] Key type .llcrypt unregistered [ 873.066544] Key type ._llcrypt unregistered [ 878.598935] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 878.712340] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 878.722140] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 878.734063] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 881.370276] Key type ._llcrypt registered [ 881.372937] Key type .llcrypt registered [ 881.539619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 881.556155] alg: No test for adler32 (adler32-zlib) [ 882.461215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.521085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 882.560516] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 883.307484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.326252] LNet: 46352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.332707] LNetError: 46352:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.352424] LNet: Removed LNI 192.168.203.10@tcp [ 883.382899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.454959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 883.474071] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 890.123881] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:10:28 (1781237428) [ 890.215066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.261182] LNet: 46943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.268932] LNetError: 46943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.302788] LNet: Removed LNI 192.168.203.10@tcp [ 890.332760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.514949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 896.348821] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:10:35 (1781237435) [ 896.649684] LNet: 47532:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.229505] Key type .llcrypt unregistered [ 897.231871] Key type ._llcrypt unregistered [ 902.745472] Key type ._llcrypt registered [ 902.749865] Key type .llcrypt registered [ 902.983327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 902.995800] alg: No test for adler32 (adler32-zlib) [ 903.948097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.030548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 904.051265] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 904.796276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 904.817928] LNet: 47930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.829941] LNetError: 47930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.862291] LNet: Removed LNI 192.168.203.10@tcp [ 904.896475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 904.978526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 905.020977] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 912.014501] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:10:50 (1781237450) [ 912.165818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.199415] LNet: 48519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.207680] LNetError: 48519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.238676] LNet: Removed LNI 192.168.203.10@tcp [ 912.297448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.397748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 919.651343] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:10:58 (1781237458) [ 921.038961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 922.540428] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:11:01 (1781237461) [ 923.925761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 925.632718] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:11:04 (1781237464) [ 927.054291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 928.629770] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:11:07 (1781237467) [ 930.413911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 932.430891] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:11:10 (1781237470) [ 933.892648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 935.671652] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:11:14 (1781237474) [ 936.073603] LNet: 50577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.645334] Key type .llcrypt unregistered [ 936.647086] Key type ._llcrypt unregistered [ 943.009336] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 943.175365] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 943.184661] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 943.194719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 945.835192] Key type ._llcrypt registered [ 945.837542] Key type .llcrypt registered [ 946.006043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 946.019227] alg: No test for adler32 (adler32-zlib) [ 946.957361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.055640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.092546] LNet: Added LNI 192.168.203.10@tcp [8/512/0/180] [ 947.099319] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 947.102298] LNet: 50899:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 947.202839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.227729] LNet: 50918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.233727] LNetError: 50918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.239456] LNet: Removed LNI 192.168.203.10@tcp [ 947.241979] LNetError: 50918:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.251732] LNet: Removed LNI 10.1.2.3@tcp [ 947.272635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.335882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.372399] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 947.436738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.453302] LNet: 50942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.456900] LNetError: 50942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.471884] LNet: Removed LNI 192.168.203.10@tcp [ 947.492217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.565760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.605837] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 947.610437] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.617794] LNet: 50950:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 947.711171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 947.733615] LNet: 50968:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 947.741221] LNetError: 50968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 947.744663] LNet: Removed LNI 192.168.203.10@tcp [ 947.757861] LNetError: 50968:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 947.776040] LNet: Removed LNI 10.1.2.3@tcp [ 947.801469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 947.899616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 947.933840] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 947.939966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 947.943498] LNet: 50976:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.086969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.102599] LNet: 51002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.112691] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.117287] LNet: Removed LNI 192.168.203.10@tcp [ 948.119454] LNetError: 51002:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.141343] LNet: Removed LNI 10.1.2.3@tcp [ 948.160259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.249913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.272794] LNet: Added LNI 192.168.203.10@tcp [8/256/0/182] [ 948.277402] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 948.369172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.399187] LNet: 51030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.403744] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.413735] LNet: Removed LNI 192.168.203.10@tcp [ 948.417539] LNetError: 51030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 948.430613] LNet: Removed LNI 10.1.2.3@tcp [ 948.462126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.550369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 948.576652] LNet: Added LNI 192.168.203.10@tcp [8/256/0/182] [ 948.580192] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 948.582724] LNet: 51038:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 948.645963] LNet: Removed LNI 192.168.203.10@tcp [ 948.661883] LNet: Removed LNI 10.1.2.3@tcp [ 949.270061] Key type .llcrypt unregistered [ 949.272779] Key type ._llcrypt unregistered [ 961.544491] Key type ._llcrypt registered [ 961.547930] Key type .llcrypt registered [ 962.890284] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:11:41 (1781237501) [ 963.660459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 963.679236] alg: No test for adler32 (adler32-zlib) [ 964.697993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.794503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.817651] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 965.509239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.10 [ 965.527431] LNet: 52014:0:(config.c:1627:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 965.537175] LNet: Removed LNI 192.168.203.10@tcp [ 972.659388] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:11:50 (1781237510) [ 972.821169] LNet: 52581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 972.834587] LNetError: 52581:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 972.891510] LNet: Removed LNI 192.168.203.10@tcp [ 973.831396] Key type .llcrypt unregistered [ 973.834467] Key type ._llcrypt unregistered [ 980.328524] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.527770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.219156] Key type ._llcrypt registered [ 983.221417] Key type .llcrypt registered [ 983.410757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.428198] alg: No test for adler32 (adler32-zlib) [ 984.864230] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 991.677788] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:12:10 (1781237530) [ 991.991856] LNet: 53495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.006444] LNetError: 53495:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.046772] LNet: Removed LNI 10.1.2.3@tcp [ 992.739695] Key type .llcrypt unregistered [ 992.750676] Key type ._llcrypt unregistered [ 999.738862] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.998546] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1000.007362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1000.011957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.744177] Key type ._llcrypt registered [ 1002.752468] Key type .llcrypt registered [ 1002.877855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.894848] alg: No test for adler32 (adler32-zlib) [ 1003.958037] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1010.937624] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:12:29 (1781237549) [ 1012.918951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1014.984335] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:12:33 (1781237553) [ 1015.074427] LNet: 54671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.083353] LNetError: 54671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1015.106792] LNet: Removed LNI 10.1.2.3@tcp [ 1016.104595] Key type .llcrypt unregistered [ 1016.110631] Key type ._llcrypt unregistered [ 1023.246387] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1023.502298] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1023.509329] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1023.518405] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1026.170343] Key type ._llcrypt registered [ 1026.172956] Key type .llcrypt registered [ 1026.406722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1026.444087] alg: No test for adler32 (adler32-zlib) [ 1027.331658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.441878] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1037.573945] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:12:56 (1781237576) [ 1038.304685] LNet: 55914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.320573] LNetError: 55914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1038.343485] LNet: Removed LNI 10.1.2.3@tcp [ 1039.291556] Key type .llcrypt unregistered [ 1039.294048] Key type ._llcrypt unregistered [ 1045.307658] Key type ._llcrypt registered [ 1045.311646] Key type .llcrypt registered [ 1045.427151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1045.436184] alg: No test for adler32 (adler32-zlib) [ 1046.320651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1046.339082] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1054.468180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1055.124557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_interrupt -r 1 [ 1055.173937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_interrupt -r 1 [ 1055.218510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1055.243375] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1055.467749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.514160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1055.564922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.585324] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1055.604880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.641494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.703388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.755833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1055.828986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1055.863325] LNet: 56352:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1055.903767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1055.935682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1055.990640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.037720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.113677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.150779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.189901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1056.782814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_dropped -r 1 [ 1056.850674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_dropped -r 1 [ 1056.896431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1056.932311] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1057.268756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.306173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1057.342748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.387566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.439187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.485317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.534735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.582292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.611706] LNet: 56462:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.639403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.676788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.705212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.740631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.770993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.797290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.822808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1058.350962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_aborted -r 1 [ 1058.407103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_aborted -r 1 [ 1058.466709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1058.491138] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1058.500697] LNet: Skipped 1 previous similar message [ 1058.739732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1058.776411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1058.816674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1058.859563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1058.901789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1058.941441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1058.981832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.050898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.097414] LNet: 56571:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1059.118884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1059.141735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1059.166756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1059.213805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1059.257374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1059.290752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1059.340260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1059.825237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_no_route -r 1 [ 1059.863517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_no_route -r 1 [ 1059.915905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1060.221208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1060.281210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1060.321906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1060.374555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1060.412822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1060.444537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1060.470772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1060.503870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1060.546024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1060.581424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1060.611364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1060.645320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1060.680419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1060.711807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1060.744591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1061.386653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_timeout -r 1 [ 1061.432491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_timeout -r 1 [ 1061.496684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1061.532753] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1061.537644] LNet: Skipped 1 previous similar message [ 1061.767894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1061.794426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1061.829764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1061.856435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1061.900413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1061.939951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1061.971711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1062.021180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1062.051392] LNet: 56790:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1062.060296] LNet: 56790:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1062.095107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1062.157769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1062.226928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1062.273882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1062.313376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1062.354341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1062.380529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1062.945324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_error -r 1 [ 1062.990615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_error -r 1 [ 1063.042091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1063.369526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.422692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1063.477707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.524420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.570934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.628111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.673957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.727513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.760709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.821317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.868946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.932707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.986894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1064.028800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1064.059217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1064.404142] LNet: 56940:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1064.420970] LNetError: 56940:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1064.445740] LNet: Removed LNI 192.168.203.10@tcp [ 1064.856498] Key type .llcrypt unregistered [ 1064.858203] Key type ._llcrypt unregistered [ 1081.668336] Key type ._llcrypt registered [ 1081.671497] Key type .llcrypt registered [ 1082.926980] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:13:41 (1781237621) [ 1083.835319] Key type .llcrypt unregistered [ 1083.840919] Key type ._llcrypt unregistered [ 1089.665640] Key type ._llcrypt registered [ 1089.670256] Key type .llcrypt registered [ 1089.860527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1089.878109] alg: No test for adler32 (adler32-zlib) [ 1090.838541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1090.884867] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1098.574728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1108.550686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1108.617552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1108.647484] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1110.103459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_interrupt -r 1 [ 1110.130637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_interrupt -r 1 [ 1110.155833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_interrupt -r 1 [ 1110.187897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_interrupt -r 1 [ 1110.241195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_interrupt -r 1 [ 1110.276387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_interrupt -r 1 [ 1110.314334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_interrupt -r 1 [ 1110.380848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_interrupt -r 1 [ 1110.435920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1110.459175] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1110.465230] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1110.676314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.710612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1110.749347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.779659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.822414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.890895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.945201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.002204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.028740] LNet: 58407:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.047654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.087899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.139195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.202125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1111.251721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1111.286234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.338663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1111.920416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_dropped -r 1 [ 1111.970405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_dropped -r 1 [ 1111.996650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_dropped -r 1 [ 1112.034745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_dropped -r 1 [ 1112.094108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_dropped -r 1 [ 1112.144909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_dropped -r 1 [ 1112.195427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_dropped -r 1 [ 1112.259776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_dropped -r 1 [ 1112.336527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1112.360375] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1112.378106] LNet: Skipped 2 previous similar messages [ 1112.670474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1112.705912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1112.742496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1112.781143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.826936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.882576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.953800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1113.009503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1113.038455] LNet: 58529:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.073132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1113.111404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1113.173121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1113.247896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1113.287844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1113.332123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1113.375202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1113.867258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_aborted -r 1 [ 1113.927915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_aborted -r 1 [ 1113.978955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_aborted -r 1 [ 1114.034626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_aborted -r 1 [ 1114.093117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_aborted -r 1 [ 1114.126379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_aborted -r 1 [ 1114.176379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_aborted -r 1 [ 1114.218754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_aborted -r 1 [ 1114.296464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1114.319397] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1114.333539] LNet: Skipped 2 previous similar messages [ 1114.591871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1114.636369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1114.658609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.684815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.720306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.751699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.789711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.826573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.848944] LNet: 58650:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1114.868297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.901421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.937909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.968256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.998554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1115.021630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1115.056469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1115.605711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_no_route -r 1 [ 1115.660103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_no_route -r 1 [ 1115.707487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_no_route -r 1 [ 1115.766322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_no_route -r 1 [ 1115.817244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_no_route -r 1 [ 1115.874964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_no_route -r 1 [ 1115.915534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_no_route -r 1 [ 1115.964689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_no_route -r 1 [ 1116.027241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1116.270765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.308319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1116.338271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.370777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1116.428798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1116.476363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1116.531462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1116.575941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1116.605996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1116.627925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1116.653428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1116.684709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1116.718341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.772417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.803957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.287975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_timeout -r 1 [ 1117.352928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_timeout -r 1 [ 1117.395600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_timeout -r 1 [ 1117.431817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_timeout -r 1 [ 1117.464581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_timeout -r 1 [ 1117.501678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_timeout -r 1 [ 1117.539509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_timeout -r 1 [ 1117.580286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_timeout -r 1 [ 1117.617938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1117.652493] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1117.656480] LNet: Skipped 5 previous similar messages [ 1117.899404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1117.936956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1117.970543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.001344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.037569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.086338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.114447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.149397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.169225] LNet: 58893:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.174882] LNet: 58893:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1118.196357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.238531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.292931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.331139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.378780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.424912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.465587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1119.036677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_error -r 1 [ 1119.098207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_error -r 1 [ 1119.146924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_error -r 1 [ 1119.200517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_error -r 1 [ 1119.285631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_error -r 1 [ 1119.346926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_error -r 1 [ 1119.403033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_error -r 1 [ 1119.450898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_error -r 1 [ 1119.503362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1119.769820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.797165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1119.826526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.867677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.921534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.957611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.978580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.002237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.022355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1120.053151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1120.077270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1120.125769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1120.192588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1120.221781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1120.262539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1125.776714] LNet: 59090:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.786737] LNetError: 59090:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.794457] LNet: Removed LNI 192.168.203.10@tcp [ 1125.806330] LNet: Removed LNI 192.168.203.10@tcp1 [ 1126.289987] Key type .llcrypt unregistered [ 1126.300615] Key type ._llcrypt unregistered [ 1142.809670] Key type ._llcrypt registered [ 1142.812261] Key type .llcrypt registered [ 1144.413041] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:14:42 (1781237682) [ 1145.280298] Key type .llcrypt unregistered [ 1145.282307] Key type ._llcrypt unregistered [ 1152.225748] Key type ._llcrypt registered [ 1152.229260] Key type .llcrypt registered [ 1152.406180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1152.421543] alg: No test for adler32 (adler32-zlib) [ 1153.380693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1153.431512] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1161.800603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1162.573968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_dropped -r 1 [ 1162.620602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_dropped -r 1 [ 1162.654343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1162.688290] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1162.960811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.012099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1163.047934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.107937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.155362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.202852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.243891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.303679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.336993] LNet: 60341:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.382977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.432818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.499913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.553820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.610938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.656808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.729239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.301657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_error -r 1 [ 1164.361860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_error -r 1 [ 1164.411638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1164.434497] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1164.688085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.720866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1164.755827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1164.811608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1164.861593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1164.913527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1164.976722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.019985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.058351] LNet: 60451:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.104174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.156169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.209349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.241905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.279560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.312464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.356942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1165.905936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_timeout -r 1 [ 1165.953773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_timeout -r 1 [ 1165.992223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1166.015936] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1166.232943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.264671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1166.288295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.314735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.337833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.362777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.398816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.442806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.464284] LNet: 60560:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.481323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.515440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.555573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.587580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.624527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.672567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.730743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.061725] LNet: 60601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.086039] LNetError: 60601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.117163] LNet: Removed LNI 192.168.203.10@tcp [ 1167.644713] Key type .llcrypt unregistered [ 1167.647677] Key type ._llcrypt unregistered [ 1183.202832] Key type ._llcrypt registered [ 1183.206689] Key type .llcrypt registered [ 1184.549416] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:15:23 (1781237723) [ 1185.332592] Key type .llcrypt unregistered [ 1185.336199] Key type ._llcrypt unregistered [ 1191.202767] Key type ._llcrypt registered [ 1191.207681] Key type .llcrypt registered [ 1191.543691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1191.573158] alg: No test for adler32 (adler32-zlib) [ 1192.468518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1192.501766] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1200.316739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1209.652600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1209.707917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1209.753815] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1211.434472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_dropped -r 1 [ 1211.483209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e remote_dropped -r 1 [ 1211.542181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_dropped -r 1 [ 1211.587315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e remote_dropped -r 1 [ 1211.629818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e remote_dropped -r 1 [ 1211.675804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e remote_dropped -r 1 [ 1211.726525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e remote_dropped -r 1 [ 1211.790863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e remote_dropped -r 1 [ 1211.853083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1211.876431] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1211.880192] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.110@tcp [ 1212.124897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1212.174848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1212.212738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1212.281515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1212.319493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1212.361393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1212.423506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1212.474694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1212.514433] LNet: 62069:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1212.547493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1212.596299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1212.643727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1212.695994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.742996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1212.811109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1212.868853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1212.924378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1213.434287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_error -r 1 [ 1213.466570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e remote_error -r 1 [ 1213.504610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_error -r 1 [ 1213.538661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e remote_error -r 1 [ 1213.586964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e remote_error -r 1 [ 1213.632603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e remote_error -r 1 [ 1213.683190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e remote_error -r 1 [ 1213.744606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e remote_error -r 1 [ 1213.825953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1213.877801] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1213.885979] LNet: Skipped 2 previous similar messages [ 1214.203719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.242221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1214.289823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.348760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.400429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.448679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.524117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.577438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.612786] LNet: 62192:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.628881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.663772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.719699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.777710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.803797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.838304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.886316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1214.945262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.589527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_timeout -r 1 [ 1215.632649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e remote_timeout -r 1 [ 1215.688274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_timeout -r 1 [ 1215.748550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e remote_timeout -r 1 [ 1215.808149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e remote_timeout -r 1 [ 1215.860685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e remote_timeout -r 1 [ 1215.912793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e remote_timeout -r 1 [ 1216.007287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e remote_timeout -r 1 [ 1216.055689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.110@tcp [ 1216.073334] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1216.380436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1216.405149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1216.430445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1216.454716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1216.480376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1216.508914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1216.536459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.556598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.569287] LNet: 62316:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.580493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.601687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.623364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.644632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.715657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.748473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.794880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1216.812174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.894911] LNet: 62394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.913181] LNetError: 62394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.919508] LNet: Removed LNI 192.168.203.10@tcp [ 1221.939497] LNet: Removed LNI 192.168.203.10@tcp1 [ 1222.345899] Key type .llcrypt unregistered [ 1222.348336] Key type ._llcrypt unregistered [ 1238.802723] Key type ._llcrypt registered [ 1238.804788] Key type .llcrypt registered [ 1240.220126] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:16:18 (1781237778) [ 1241.686225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1243.652633] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:16:22 (1781237782) [ 1244.545372] Key type .llcrypt unregistered [ 1244.547326] Key type ._llcrypt unregistered [ 1251.659469] Key type ._llcrypt registered [ 1251.671766] Key type .llcrypt registered [ 1251.950495] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1251.970322] alg: No test for adler32 (adler32-zlib) [ 1252.960914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1252.999070] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1261.424880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1262.362332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e network_timeout -r 1 [ 1262.416737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e network_timeout -r 1 [ 1262.455899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1262.475134] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1262.755251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1262.798409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1262.836258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1262.880986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1262.962664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.032415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.090989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.132697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.160966] LNet: 63938:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.183489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.217767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.263698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.268484] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1263.275081] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1263.319482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.347480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.380386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.683110] LNet: 63977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.707691] LNetError: 63977:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1263.733041] LNet: Removed LNI 192.168.203.10@tcp [ 1264.226436] Key type .llcrypt unregistered [ 1264.230460] Key type ._llcrypt unregistered [ 1283.022841] Key type ._llcrypt registered [ 1283.027806] Key type .llcrypt registered [ 1283.233370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1283.248463] alg: No test for adler32 (adler32-zlib) [ 1284.209151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1284.235082] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1291.982320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1302.057664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.133754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1302.178642] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1303.643155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e network_timeout -r 1 [ 1303.677728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e network_timeout -r 1 [ 1303.723634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e network_timeout -r 1 [ 1303.766515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e network_timeout -r 1 [ 1303.803534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e network_timeout -r 1 [ 1303.886556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e network_timeout -r 1 [ 1303.939946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e network_timeout -r 1 [ 1303.965830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e network_timeout -r 1 [ 1303.993234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1304.004890] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1304.178981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1304.225119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1304.269702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.311727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.347258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.386806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.422604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.458553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.479214] LNet: 64856:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1304.502624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.563778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.609719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.644974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.680610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.717044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1309.942071] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1310.265852] LNet: 64930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.275521] LNetError: 64930:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.284690] LNet: Removed LNI 192.168.203.10@tcp [ 1310.301337] LNet: Removed LNI 192.168.203.10@tcp1 [ 1310.836592] Key type .llcrypt unregistered [ 1310.840070] Key type ._llcrypt unregistered [ 1329.691763] Key type ._llcrypt registered [ 1329.698847] Key type .llcrypt registered [ 1331.852352] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:17:49 (1781237869) [ 1332.455736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.481938] alg: No test for adler32 (adler32-zlib) [ 1333.565727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1333.695625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1333.753474] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1334.786674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1334.809053] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1335.827127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 1335.942901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1336.015564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1336.078849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1336.152756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1336.213806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1336.313815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.10@tcp [ 1336.370861] LNetError: 66045:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.10@tcp to recovery [ 1336.672442] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1336.679502] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1338.722405] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1339.023855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1342.818146] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1343.887737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1343.923736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.000959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1344.041824] LNet: 66153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.059922] LNetError: 66153:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.075926] LNet: Removed LNI 192.168.203.10@tcp [ 1344.104212] LNet: Removed LNI 192.168.203.10@tcp1 [ 1344.148834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.205347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1344.230283] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1344.897443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.925920] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1345.693861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 1345.759924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1345.822628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1345.921808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1345.987676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1346.049727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1346.105981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1346.149831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.10@tcp [ 1346.190737] LNetError: 66425:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.10@tcp to recovery [ 1346.275867] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1348.554503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1352.419041] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1352.432530] LNet: Skipped 1 previous similar message [ 1357.624471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1357.664401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1357.724443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1365.328884] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:18:23 (1781237903) [ 1365.554705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1365.581634] LNet: 67143:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1365.587913] LNetError: 67143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1365.594393] LNetError: 67143:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1365.604528] LNet: Removed LNI 192.168.203.10@tcp [ 1365.639360] LNet: Removed LNI 192.168.203.10@tcp1 [ 1365.695518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1365.771906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1365.823593] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1366.560754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1366.591678] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1367.584723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 1367.678466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1367.709433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1367.743456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1367.803697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1367.846978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1367.882438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.10@tcp --health 0 [ 1367.993506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1368.800282] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.10@tcp [ 1368.800870] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1368.828690] LNet: Skipped 1 previous similar message [ 1382.614518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1382.779506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1382.811360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1382.847643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.10@tcp --health 500 [ 1383.047933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.735285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1388.779624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1388.793063] LNet: 67618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1388.809983] LNetError: 67618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.814323] LNetError: 67618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1388.821305] LNet: Removed LNI 192.168.203.10@tcp [ 1388.831921] LNet: Removed LNI 192.168.203.10@tcp1 [ 1388.862795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.939598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1388.981789] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1390.176470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1390.190116] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1391.083651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 1391.143884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.239662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1391.276729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1391.335854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1391.361513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1391.388159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1391.415140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.10@tcp --health 0 [ 1392.992561] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1393.002934] LNet: Skipped 3 previous similar messages [ 1393.936802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.177425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1403.202038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1403.225569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.386368] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:19:08 (1781237948) [ 1410.758069] LNet: 68665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.779896] LNetError: 68665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.786604] LNetError: 68665:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1410.790093] LNet: Removed LNI 192.168.203.10@tcp [ 1410.801789] LNet: Removed LNI 192.168.203.10@tcp1 [ 1411.506577] Key type .llcrypt unregistered [ 1411.509269] Key type ._llcrypt unregistered [ 1417.644716] Key type ._llcrypt registered [ 1417.651529] Key type .llcrypt registered [ 1417.838046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.850923] alg: No test for adler32 (adler32-zlib) [ 1418.733584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1418.754684] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1427.747914] Lustre: DEBUG MARKER: Initial discovery [ 1427.930921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.110@tcp [ 1430.257854] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1430.619908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1430.691808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.110@tcp [ 1430.746792] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1430.776403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1430.833983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.110@tcp -r 1 -m GET -l 3 [ 1430.898531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.110@tcp -r 1 -m GET -e local_error [ 1430.973615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.110@tcp -r 1 -m PUT -l 6 [ 1431.074785] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1432.401322] Lustre: DEBUG MARKER: Force oleg310-client.virtnet to discover 192.168.203.110@tcp (in background) [ 1432.484542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.110@tcp [ 1434.272664] Lustre: DEBUG MARKER: Force oleg310-server to discover 192.168.203.10@tcp [ 1435.552761] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1436.703773] Lustre: DEBUG MARKER: Wait for 69420 [ 1438.146856] Lustre: DEBUG MARKER: Finished wait on 69420 [ 1438.208116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1438.246840] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1438.260280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1438.732138] LNet: 69879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1438.749713] LNetError: 69879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1439.784304] LNet: Removed LNI 192.168.203.10@tcp [ 1440.187759] Key type .llcrypt unregistered [ 1440.193139] Key type ._llcrypt unregistered [ 1455.584040] Key type ._llcrypt registered [ 1455.585898] Key type .llcrypt registered [ 1456.895577] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:19:55 (1781237995) [ 1457.702509] Key type .llcrypt unregistered [ 1457.708711] Key type ._llcrypt unregistered [ 1463.084518] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1463.305985] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1463.309378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1463.312626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1465.855981] Key type ._llcrypt registered [ 1465.865054] Key type .llcrypt registered [ 1466.099500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.112830] alg: No test for adler32 (adler32-zlib) [ 1467.027556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.128047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1467.163828] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1467.935242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1467.970601] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1476.567668] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:20:15 (1781238015) [ 1476.766090] LNet: 71954:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.787794] LNetError: 71954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.791661] LNet: Removed LNI 192.168.203.10@tcp [ 1476.812424] LNet: Removed LNI 10.1.2.3@tcp [ 1477.481398] Key type .llcrypt unregistered [ 1477.483398] Key type ._llcrypt unregistered [ 1483.532534] Key type ._llcrypt registered [ 1483.535631] Key type .llcrypt registered [ 1483.729079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1483.746186] alg: No test for adler32 (adler32-zlib) [ 1484.764502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1484.839125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1484.869928] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1485.674586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1485.693936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1486.774447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.10@tcp [ 1487.850300] LNetError: Unexpected error -22 connecting to 192.168.203.10@tcp at host 192.168.203.10:988 [ 1493.477520] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:20:31 (1781238031) [ 1493.564291] LNet: 73083:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.568632] LNetError: 73083:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.584403] LNet: Removed LNI 192.168.203.10@tcp [ 1493.598475] LNet: Removed LNI 10.1.2.3@tcp [ 1494.342435] Key type .llcrypt unregistered [ 1494.346596] Key type ._llcrypt unregistered [ 1500.251862] Key type ._llcrypt registered [ 1500.261496] Key type .llcrypt registered [ 1500.520888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1500.551844] alg: No test for adler32 (adler32-zlib) [ 1501.499508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.566724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1501.608301] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1502.557637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1502.587881] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 1503.557801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.10@tcp1 --nid 192.168.203.10@tcp2 [ 1508.988184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp1 192.168.203.10@tcp2 [ 1509.012796] LNetError: 73926:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.10@tcp1 to 192.168.203.10@tcp2 - Route table may be misconfigured [ 1509.046994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp2 192.168.203.10@tcp1 [ 1515.556341] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:20:54 (1781238054) [ 1515.977851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1516.009392] LNet: 74499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.018420] LNetError: 74499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.024685] LNet: Removed LNI 192.168.203.10@tcp1 [ 1517.044421] LNetError: 74499:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1517.078894] LNet: Removed LNI 192.168.203.10@tcp2 [ 1517.121580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1517.195599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1517.228189] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1518.049492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1518.088349] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1519.089299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 1519.127708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.10@tcp [ 1519.165225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp1 192.168.203.10@tcp1 [ 1519.213232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.10@tcp -d 192.168.203.10@tcp -e network_timeout [ 1519.251232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -e network_timeout [ 1519.306344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -e network_timeout [ 1519.358438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -e network_timeout [ 1519.431396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1519.455707] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1519.486306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1519.598199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1519.696705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1520.224258] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1526.668959] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:21:05 (1781238065) [ 1527.036963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1527.074079] LNet: 75362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1527.085216] LNetError: 75362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1527.097392] LNet: Removed LNI 192.168.203.10@tcp [ 1528.167435] LNet: Removed LNI 192.168.203.10@tcp1 [ 1528.241869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.381617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1528.680106] LNet: 75399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.193896] Key type .llcrypt unregistered [ 1529.195909] Key type ._llcrypt unregistered [ 1541.700756] Key type ._llcrypt registered [ 1541.704780] Key type .llcrypt registered [ 1543.030570] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:21:21 (1781238081) [ 1543.588551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.618803] alg: No test for adler32 (adler32-zlib) [ 1544.559123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.957965] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1547.798360] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:21:26 (1781238086) [ 1548.121593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.153983] LNet: 76496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.170273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1548.232553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1548.258928] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1549.117633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1549.144916] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1550.189749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1550.230266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp1 [ 1550.265056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp1 [ 1556.525109] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:21:35 (1781238095) [ 1558.309210] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg310-server [ 1560.368847] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:21:38 (1781238098) [ 1562.618578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg310-server [ 1564.475724] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:21:42 (1781238102) [ 1566.768687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg310-server [ 1568.647802] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:21:47 (1781238107) [ 1570.508679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg310-server [ 1572.088356] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:21:50 (1781238110) [ 1573.779361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg310-server [ 1575.332817] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:21:54 (1781238114) [ 1577.146760] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg310-server [ 1579.106974] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:21:57 (1781238117) [ 1580.713775] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg310-server [ 1582.240984] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:22:01 (1781238121) [ 1583.751924] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg310-server [ 1585.075782] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:22:04 (1781238124) [ 1586.802067] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg310-server [ 1588.280679] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:22:07 (1781238127) [ 1588.433453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.459422] LNet: 80373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.464629] LNetError: 80373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.472333] LNet: Removed LNI 192.168.203.10@tcp [ 1589.474826] LNetError: 80373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.496924] LNet: Removed LNI 192.168.203.10@tcp1 [ 1589.535145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.611426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.637379] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1590.379406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1590.471848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1590.750209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1590.788322] LNet: 80521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.793377] LNetError: 80521:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1590.817522] LNet: Removed LNI 192.168.203.10@tcp [ 1590.869317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.942718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1590.976165] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1591.849597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1592.113269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.193584] LNet: Removed LNI 192.168.203.10@tcp [ 1593.231329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.295211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1593.329464] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1594.044429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1594.113200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1594.314855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.345796] LNet: 80813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.353391] LNet: 80813:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1594.362211] LNetError: 80813:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.369383] LNetError: 80813:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1594.386427] LNet: Removed LNI 192.168.203.10@tcp [ 1594.404979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.478141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1594.496434] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1595.109626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1595.253628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.330487] LNet: Removed LNI 192.168.203.10@tcp [ 1596.351548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1596.418643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1596.457610] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1597.082845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1597.152348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1597.320246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.374733] LNet: Removed LNI 192.168.203.10@tcp [ 1598.403811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.463968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1598.501849] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1599.119310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1599.296813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.315592] LNet: 81251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.322463] LNet: 81251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1599.329167] LNetError: 81251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.335739] LNetError: 81251:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1599.346330] LNet: Removed LNI 192.168.203.10@tcp [ 1599.378773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1599.447352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1599.471350] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1600.131300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1600.218837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1601.571812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1602.665764] LNet: Removed LNI 192.168.203.10@tcp [ 1602.709818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.763742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1602.797278] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1603.694237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1603.952869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.034699] LNet: Removed LNI 192.168.203.10@tcp [ 1605.076206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.183120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.211161] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1605.950128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1606.070934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1606.337577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.398928] LNet: Removed LNI 192.168.203.10@tcp [ 1607.441597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.514514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1607.542924] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1608.330329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1609.874498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1609.898422] LNet: 81854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1609.906916] LNet: 81854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1609.920599] LNetError: 81854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1609.933518] LNetError: 81854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1609.952634] LNet: Removed LNI 192.168.203.10@tcp [ 1609.967469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1610.028784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1610.084990] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1610.898493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1611.005815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1611.250049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1612.334546] LNet: Removed LNI 192.168.203.10@tcp [ 1612.368819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1612.405721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1612.426263] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1613.192402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1613.584401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1614.704815] LNet: Removed LNI 192.168.203.10@tcp [ 1614.749408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.802382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1614.826350] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1615.541520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1615.597524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.618693] LNet: Removed LNI 192.168.203.10@tcp [ 1615.632399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.710867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.758552] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1616.557733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1623.037449] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:22:41 (1781238161) [ 1623.299543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1623.353882] LNet: Removed LNI 192.168.203.10@tcp [ 1623.402499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.478529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1623.527036] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 1624.823837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.849949] LNet: Removed LNI 192.168.203.10@tcp231 [ 1624.874144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.950748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1624.975241] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 1624.983182] LNet: 83130:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1625.495973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1625.535693] LNet: Removed LNI 192.168.203.10@tcp231 [ 1625.549593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1625.578882] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 1626.740532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1626.760501] LNet: 83274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1626.772177] LNet: 83274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1626.781231] LNetError: 83274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1626.787871] LNetError: 83274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1626.814381] LNet: Removed LNI 192.168.203.10@tcp231 [ 1626.839583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.930428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1626.961930] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/0] [ 1626.975717] LNet: 83282:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1627.417578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1627.458414] LNet: Removed LNI 192.168.203.10@tcp231 [ 1627.485160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1627.503503] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/0] [ 1628.535262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1628.582180] LNet: Removed LNI 192.168.203.10@tcp231 [ 1628.616547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1628.693291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1628.731452] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/60] [ 1628.740884] LNet: 83433:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1629.218481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1629.253791] LNet: Removed LNI 192.168.203.10@tcp231 [ 1629.270803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1629.306652] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/60] [ 1630.543347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1630.616327] LNet: Removed LNI 192.168.203.10@tcp231 [ 1630.644529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.712555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1630.741820] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 1630.749852] LNet: 83585:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1631.218604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1631.244433] LNet: Removed LNI 192.168.203.10@tcp231 [ 1631.266558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1631.288861] LNet: Added LNI 192.168.203.10@tcp231 [8/256/0/180] [ 1638.351170] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:22:57 (1781238177) [ 1638.494525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1638.541215] LNet: Removed LNI 192.168.203.10@tcp231 [ 1638.557709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1638.614978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1638.640788] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 1639.344683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1639.390134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1639.424101] LNet: Removed LNI 192.168.203.10@tcp232 [ 1639.439029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1639.491420] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 1640.268204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1640.326333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1640.356269] LNet: Removed LNI 192.168.203.10@tcp232 [ 1640.368117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1640.389713] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 1641.068560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1641.100923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1641.126774] LNet: Removed LNI 192.168.203.10@tcp232 [ 1641.145787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1641.201272] LNet: Added LNI 192.168.203.10@tcp232 [8/256/0/180] [ 1642.003219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1642.062456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1642.108075] LNet: Removed LNI 192.168.203.10@tcp232 [ 1642.131968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1648.221315] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:23:06 (1781238186) [ 1649.336419] Key type .llcrypt unregistered [ 1649.340112] Key type ._llcrypt unregistered [ 1655.364886] Key type ._llcrypt registered [ 1655.368030] Key type .llcrypt registered [ 1655.597532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1655.627883] alg: No test for adler32 (adler32-zlib) [ 1656.576698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1656.611229] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1663.953320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1672.555618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.614244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1672.650687] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1674.172961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_interrupt -r 2 [ 1674.203349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_interrupt -r 2 [ 1674.235177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_interrupt -r 2 [ 1674.274404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_interrupt -r 2 [ 1674.308142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_interrupt -r 2 [ 1674.348969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_interrupt -r 2 [ 1674.381266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_interrupt -r 2 [ 1674.411912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_interrupt -r 2 [ 1674.443983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.110@tcp [ 1674.463472] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1674.468407] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1674.665870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1674.697746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1674.729260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1674.762706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1674.814307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1674.843684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1674.881052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1674.921804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1674.935301] LNet: 86084:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1674.953808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1674.983536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1675.028124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1675.061710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1675.103973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1675.144745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1675.671833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_dropped -r 2 [ 1675.723067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_dropped -r 2 [ 1675.764324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_dropped -r 2 [ 1675.794328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_dropped -r 2 [ 1675.825949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_dropped -r 2 [ 1675.857287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_dropped -r 2 [ 1675.895776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_dropped -r 2 [ 1675.925605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_dropped -r 2 [ 1675.950404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.110@tcp [ 1675.968246] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1676.250874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1676.282221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1676.314300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1676.340690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1676.367660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1676.393931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1676.427274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1676.457674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1676.480479] LNet: 86204:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1676.506941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1676.542687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1676.583182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1676.613315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1676.639631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1676.671837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1677.161824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_aborted -r 2 [ 1677.206212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_aborted -r 2 [ 1677.236104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_aborted -r 2 [ 1677.268307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_aborted -r 2 [ 1677.294632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_aborted -r 2 [ 1677.334501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_aborted -r 2 [ 1677.367117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_aborted -r 2 [ 1677.426097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_aborted -r 2 [ 1677.479316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.110@tcp [ 1677.505978] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1677.783310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1677.832218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1677.873729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1677.926823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1677.973755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1678.002895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1678.054848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1678.097120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1678.119505] LNet: 86325:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1678.138646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1678.170653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1678.202817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1678.229379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1678.266466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1678.303299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1678.793388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_no_route -r 2 [ 1678.822883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_no_route -r 2 [ 1678.863759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_no_route -r 2 [ 1678.887745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_no_route -r 2 [ 1678.914434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_no_route -r 2 [ 1678.958943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_no_route -r 2 [ 1678.994887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_no_route -r 2 [ 1679.026673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_no_route -r 2 [ 1679.052441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.110@tcp [ 1679.281963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1679.309310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1679.336359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1679.371417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1679.405526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1679.436829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1679.480925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1679.508350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1679.540343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1679.587926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1679.615953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1679.640214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1679.673472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1679.707703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1680.250644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e local_timeout -r 2 [ 1680.294642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e local_timeout -r 2 [ 1680.340490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e local_timeout -r 2 [ 1680.380804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e local_timeout -r 2 [ 1680.433959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e local_timeout -r 2 [ 1680.471779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e local_timeout -r 2 [ 1680.501876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e local_timeout -r 2 [ 1680.541411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e local_timeout -r 2 [ 1680.576613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.110@tcp [ 1680.605466] LNet: There was an unexpected network error while writing to 192.168.203.110: rc = -22 [ 1680.610255] LNet: Skipped 1 previous similar message [ 1680.839236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1680.883059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1680.916109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1680.938888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1680.964533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1680.997846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1681.028499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1681.063679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1681.087678] LNet: 86565:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1681.097417] LNet: 86565:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1681.117799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1681.147262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1681.170359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1681.195621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1681.229682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1681.270598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1681.704859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp -m GET -e remote_dropped -r 2 [ 1681.739467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.110@tcp1 -m GET -e remote_dropped -r 2 [ 1681.767663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp -m GET -e remote_dropped -r 2 [ 1681.807964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp -d 192.168.203.10@tcp1 -m GET -e remote_dropped -r 2 [ 1681.838418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp -m GET -e remote_dropped -r 2 [ 1681.877921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.110@tcp1 -m GET -e remote_dropped -r 2 [ 1681.932465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp -m GET -e remote_dropped -r 2 [ 1681.994732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.10@tcp1 -d 192.168.203.10@tcp1 -m GET -e remote_dropped -r 2 [ 1682.047832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.10@tcp 192.168.203.110@tcp [ 1682.080223] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.110@tcp [ 1682.321130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1682.352925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1682.390605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1682.436857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1682.466184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1682.495139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1682.530347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1682.565024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1682.617326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1682.664132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1682.704827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1682.740783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1682.780645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1682.824355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1688.605916] LNet: 86759:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1688.614784] LNetError: 86759:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1688.620673] LNet: Removed LNI 192.168.203.10@tcp [ 1688.630435] LNet: Removed LNI 192.168.203.10@tcp1 [ 1689.100805] Key type .llcrypt unregistered [ 1689.105593] Key type ._llcrypt unregistered [ 1707.700285] Key type ._llcrypt registered [ 1707.701890] Key type .llcrypt registered [ 1709.226672] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:24:07 (1781238247) [ 1710.043352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1710.094519] alg: No test for adler32 (adler32-zlib) [ 1711.124616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.251585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1711.285289] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1713.159135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1713.193628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.10@tcp [ 1713.221783] LNetError: 87738:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.10@tcp to recovery [ 1713.256227] LNet: 1 local NIs in recovery (showing 1): 192.168.203.10@tcp [ 1713.395637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.10@tcp [ 1713.655506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1721.796656] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:24:20 (1781238260) [ 1721.908037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.941564] LNet: 88324:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.946501] LNetError: 88324:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.987825] LNet: Removed LNI 192.168.203.10@tcp [ 1723.014791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.078660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.116912] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1723.803371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1723.818714] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 1724.656212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.10@tcp [ 1724.671220] LNetError: 88594:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.203.10@tcp to recovery [ 1725.089587] LNet: There was an unexpected network error while writing to 192.168.203.10: rc = -22 [ 1725.862977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.10@tcp [ 1735.078601] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:24:33 (1781238273) [ 1737.824978] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg310-server [ 1739.975910] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:24:38 (1781238278) [ 1742.313727] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg310-server [ 1744.251590] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:24:42 (1781238282) [ 1747.100826] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg310-server [ 1749.274678] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:24:47 (1781238287) [ 1749.539297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.582744] LNet: 90225:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1749.591409] LNetError: 90225:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.596701] LNet: Removed LNI 192.168.203.10@tcp [ 1749.616938] LNet: Removed LNI 192.168.203.10@tcp1 [ 1749.641467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.745574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1749.793977] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1750.779790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1750.857905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1750.904965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1750.994212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1751.061099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1751.138168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1757.233877] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:24:56 (1781238296) [ 1757.678201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.700845] LNet: 90951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.710428] LNetError: 90951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.715170] LNetError: 90951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1757.731075] LNet: Removed LNI 192.168.203.10@tcp [ 1757.754828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.816835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.838393] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1764.494400] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:25:02 (1781238302) [ 1764.582853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.623366] LNet: 91648:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.645058] LNetError: 91648:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.678564] LNet: Removed LNI 192.168.203.10@tcp [ 1764.709833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1764.828720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1764.895330] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1774.251457] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:25:12 (1781238312) [ 1776.841614] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1778.545594] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:25:17 (1781238317) [ 1779.239626] LNet: 92689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.258429] LNetError: 92689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.294326] LNet: Removed LNI 192.168.203.10@tcp [ 1779.920736] Key type .llcrypt unregistered [ 1779.923635] Key type ._llcrypt unregistered [ 1785.677508] Key type ._llcrypt registered [ 1785.679341] Key type .llcrypt registered [ 1785.814267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.829850] alg: No test for adler32 (adler32-zlib) [ 1786.695807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1786.722373] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1793.644656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 1798.117457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.110@tcp [ 1798.487313] LNet: 93106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.499398] LNetError: 93106:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.528135] LNet: Removed LNI 192.168.203.10@tcp [ 1799.067695] Key type .llcrypt unregistered [ 1799.069353] Key type ._llcrypt unregistered [ 1809.906263] Key type ._llcrypt registered [ 1809.907980] Key type .llcrypt registered [ 1811.256555] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1812.696642] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1814.060577] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:25:53 (1781238353) [ 1815.594868] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1817.114923] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:25:55 (1781238355) [ 1818.680828] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg310-server [ 1820.589863] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:25:59 (1781238359) [ 1823.256702] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg310-server [ 1825.533509] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:26:03 (1781238363) [ 1827.473918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1827.503621] alg: No test for adler32 (adler32-zlib) [ 1828.731306] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 1828.990755] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1830.648221] Key type lgssc registered [ 1832.282060] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1853.399717] Key type lgssc unregistered [ 1853.619726] LNet: 96282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1853.638235] LNetError: 96282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1853.674470] LNet: Removed LNI 192.168.203.10@tcp [ 1854.522751] Key type .llcrypt unregistered [ 1854.528996] Key type ._llcrypt unregistered [ 1867.614869] Key type ._llcrypt registered [ 1867.621523] Key type .llcrypt registered [ 1868.986667] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:26:47 (1781238407) [ 1869.335324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.376220] alg: No test for adler32 (adler32-zlib) [ 1870.355393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.455788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1870.522793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1870.595738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1870.639971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1876.583569] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:26:55 (1781238415) [ 1876.852291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1876.874534] LNet: 97697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1876.918100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.991070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1877.029846] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1877.839328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.885994] LNet: 97841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.898102] LNetError: 97841:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.924278] LNet: Removed LNI 192.168.203.10@tcp [ 1877.955653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.021537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1878.068802] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1878.486198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.516665] LNetError: 97869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1878.536742] LNet: Removed LNI 192.168.203.10@tcp [ 1878.562797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.615948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1878.640949] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1885.432071] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:27:04 (1781238424) [ 1885.572087] LNet: 98459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.589734] LNet: 98459:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1885.603453] LNetError: 98459:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1885.633867] LNet: Removed LNI 192.168.203.10@tcp [ 1886.315656] Key type .llcrypt unregistered [ 1886.317438] Key type ._llcrypt unregistered [ 1891.605849] Key type ._llcrypt registered [ 1891.611792] Key type .llcrypt registered [ 1891.828828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1891.850767] alg: No test for adler32 (adler32-zlib) [ 1892.741313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1892.837543] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1892.841040] LNet: *** cfs_fail_loc=e003, val=100*** [ 1893.631365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1893.650900] LNet: *** cfs_fail_loc=e003, val=100*** [ 1893.654203] LNet: Skipped 2 previous similar messages [ 1893.659784] Ping reply received with mapping: [ 1893.664678] found metadata reply NI[192.168.203.10@tcp] [ 1893.669562] 0a cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1893.673441] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1893.677317] 5a 5a 5a 5a 5a 5a 5a 5a [ 1893.835946] LNet: 98857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.845424] LNetError: 98857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.873225] LNet: Removed LNI 192.168.203.10@tcp [ 1894.423518] Key type .llcrypt unregistered [ 1894.425819] Key type ._llcrypt unregistered [ 1902.940035] Key type ._llcrypt registered [ 1902.941683] Key type .llcrypt registered [ 1904.097897] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:27:22 (1781238442) [ 1904.523297] Key type .llcrypt unregistered [ 1904.525815] Key type ._llcrypt unregistered [ 1909.747037] Key type ._llcrypt registered [ 1909.754769] Key type .llcrypt registered [ 1909.933577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1909.947158] alg: No test for adler32 (adler32-zlib) [ 1911.038145] LNetError: 99903:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1911.471282] Key type .llcrypt unregistered [ 1911.477522] Key type ._llcrypt unregistered [ 1916.540425] Key type ._llcrypt registered [ 1916.552594] Key type .llcrypt registered [ 1917.871472] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:27:36 (1781238456) [ 1918.395337] Key type .llcrypt unregistered [ 1918.397714] Key type ._llcrypt unregistered [ 1923.700859] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1923.705332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1926.288359] Key type ._llcrypt registered [ 1926.293035] Key type .llcrypt registered [ 1926.491190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1926.507496] alg: No test for adler32 (adler32-zlib) [ 1927.484882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.565156] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1938.328355] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:27:57 (1781238477) [ 1938.566587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1938.595978] LNet: 101685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.610207] LNetError: 101685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1938.634111] LNet: Removed LNI 10.1.2.3@tcp [ 1938.646414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.350138] Key type .llcrypt unregistered [ 1939.359484] Key type ._llcrypt unregistered [ 1945.384712] Key type ._llcrypt registered [ 1945.387623] Key type .llcrypt registered [ 1945.510724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1945.521849] alg: No test for adler32 (adler32-zlib) [ 1946.648703] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1946.769439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1947.055859] LNet: 101973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.076966] LNetError: 101973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.123769] LNet: Removed LNI 192.168.203.10@tcp [ 1948.049721] Key type .llcrypt unregistered [ 1948.052156] Key type ._llcrypt unregistered [ 1961.381891] Key type ._llcrypt registered [ 1961.384931] Key type .llcrypt registered [ 1963.061611] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:28:21 (1781238501) [ 1963.723411] Key type .llcrypt unregistered [ 1963.727609] Key type ._llcrypt unregistered [ 1970.406364] Key type ._llcrypt registered [ 1970.411342] Key type .llcrypt registered [ 1970.739160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1970.773898] alg: No test for adler32 (adler32-zlib) [ 1977.849429] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:28:36 (1781238516) [ 1978.041400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1978.063264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1978.115744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1978.142475] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1978.885381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1978.911913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1978.937035] LNet: Removed LNI 192.168.203.10@tcp [ 1979.202506] LNet: 103726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.599432] Key type .llcrypt unregistered [ 1979.603662] Key type ._llcrypt unregistered [ 1991.722347] Key type ._llcrypt registered [ 1991.723946] Key type .llcrypt registered [ 1992.887985] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:28:51 (1781238531) [ 1993.430128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.449089] alg: No test for adler32 (adler32-zlib) [ 1994.399123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1994.443636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1994.475442] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 1995.163371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 1995.197977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.10@tcp [ 1995.215519] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1995.222509] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1995.229356] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1995, 0, 0 [ 1995.237727] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1995.242528] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1995.247070] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1995.252705] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1995, 0, 0 [ 1995.257506] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1995.260051] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1995.264707] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1995.269859] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1995, 0, 2045 [ 1995.274445] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1995.276841] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1995.279843] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1995.282908] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1995, 0, 2045 [ 1995.287464] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2001.559491] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:29:00 (1781238540) [ 2002.275198] LNet: 105266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.283227] LNetError: 105266:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.293987] LNet: Removed LNI 192.168.203.10@tcp [ 2002.883967] Key type .llcrypt unregistered [ 2002.886958] Key type ._llcrypt unregistered [ 2009.102430] Key type ._llcrypt registered [ 2009.105418] Key type .llcrypt registered [ 2009.285507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2009.307449] alg: No test for adler32 (adler32-zlib) [ 2010.211107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2010.245527] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2017.049514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 2026.209246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2026.236625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2026.254478] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 2027.301222] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2027.366094] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2027.370035] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2027.374153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2029.808590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2029.843865] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2042.887535] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:29:41 (1781238581) [ 2043.119148] LNet: 106770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2043.148294] LNetError: 106770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2043.166836] LNet: Removed LNI 192.168.203.10@tcp [ 2044.192228] LNet: Removed LNI 192.168.203.10@tcp1 [ 2044.202239] LNetError: 106770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2044.218683] LNetError: 106770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2044.246225] LNet: Removed LNI 10.1.2.3@tcp99 [ 2045.037418] Key type .llcrypt unregistered [ 2045.041179] Key type ._llcrypt unregistered [ 2053.017961] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2053.023287] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2055.734482] Key type ._llcrypt registered [ 2055.737847] Key type .llcrypt registered [ 2055.970618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2055.987961] alg: No test for adler32 (adler32-zlib) [ 2056.931971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.972617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2056.991825] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2057.633839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2057.655461] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2059.761810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2059.828203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2059.902558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2059.923726] LNetError: 107522:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2059.961794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2060.011433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2060.539280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2060.650899] LNet: 107543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.666564] LNetError: 107543:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.674698] LNet: Removed LNI 192.168.203.10@tcp [ 2060.701280] LNet: Removed LNI 10.1.2.3@tcp [ 2061.465127] Key type .llcrypt unregistered [ 2061.467316] Key type ._llcrypt unregistered [ 2074.047179] Key type ._llcrypt registered [ 2074.051460] Key type .llcrypt registered [ 2075.286813] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:30:13 (1781238613) [ 2075.848996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.868309] alg: No test for adler32 (adler32-zlib) [ 2076.759882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.813320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.857824] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2077.734807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.10@tcp [ 2077.768504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg310-client@tcp [ 2083.272860] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:30:22 (1781238622) [ 2083.455104] LNet: 109076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.464287] LNetError: 109076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2083.479544] LNet: Removed LNI 192.168.203.10@tcp [ 2083.978607] Key type .llcrypt unregistered [ 2083.981445] Key type ._llcrypt unregistered [ 2089.240820] Key type ._llcrypt registered [ 2089.243160] Key type .llcrypt registered [ 2089.441829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2089.456435] alg: No test for adler32 (adler32-zlib) [ 2090.288787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2090.306672] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2090.331664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2096.286719] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:30:35 (1781238635) [ 2097.833340] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2099.504367] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:30:38 (1781238638) [ 2100.822591] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2102.925453] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:30:41 (1781238641) [ 2103.308564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2103.329167] LNet: 110497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.334993] LNetError: 110497:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2103.350226] LNet: Removed LNI 192.168.203.10@tcp [ 2103.371405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.462297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2103.485072] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2104.205717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2104.295845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2110.817323] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:30:49 (1781238649) [ 2111.045602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.068023] LNet: 111206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2111.072958] LNetError: 111206:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.106218] LNet: Removed LNI 192.168.203.10@tcp [ 2112.159355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2112.239518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2112.301664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2112.564955] LNet: 111242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.923326] Key type .llcrypt unregistered [ 2112.926111] Key type ._llcrypt unregistered [ 2124.453715] Key type ._llcrypt registered [ 2124.455910] Key type .llcrypt registered [ 2125.628022] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:31:04 (1781238664) [ 2126.226273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.244860] alg: No test for adler32 (adler32-zlib) [ 2127.252367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2127.352987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2127.395060] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2128.073636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2128.134113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2134.353426] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:31:12 (1781238672) [ 2134.448490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.477524] LNet: 112755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.490430] LNetError: 112755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2134.515269] LNet: Removed LNI 192.168.203.10@tcp [ 2134.555550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.658968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2134.697784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2142.256611] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:31:20 (1781238680) [ 2142.429877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2142.486540] LNet: 113338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.574588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.691940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2143.191292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2143.254637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2143.348512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2143.890725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2143.937368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2144.081926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.115565] LNet: 113378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2144.130926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.178603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2144.718715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2144.788991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2144.898483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2145.365482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2145.445234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2151.716970] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:31:30 (1781238690) [ 2151.812427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.838457] LNet: 113971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.872622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.949707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2151.981731] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2152.822080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2152.856942] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2153.890346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2153.970425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2154.163718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2154.187435] LNetError: 114258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.232137] LNet: Removed LNI 192.168.203.10@tcp [ 2155.242279] LNetError: 114258:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.267748] LNet: Removed LNI 192.168.203.10@tcp2 [ 2155.326161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.416555] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2155.424235] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2155.428987] LNet: 114264:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2155.937855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2156.143984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.162218] LNet: 114301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.167183] LNet: 114301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2157.216146] LNet: Removed LNI 192.168.203.10@tcp [ 2157.231728] LNet: Removed LNI 192.168.203.10@tcp2 [ 2157.277859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.340164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.361971] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2157.938393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2157.963509] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2158.988648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2159.062140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2159.114320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2159.355525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2159.381823] LNetError: 114586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2159.390421] LNetError: 114586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2159.396890] LNet: Removed LNI 192.168.203.10@tcp [ 2160.424630] LNet: Removed LNI 192.168.203.10@tcp2 [ 2160.462404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.544082] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2160.556267] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2160.563281] LNet: 114592:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2161.170598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2161.315515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2161.331302] LNet: Removed LNI 192.168.203.10@tcp [ 2162.408974] LNet: Removed LNI 192.168.203.10@tcp2 [ 2162.461718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.522806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2162.566289] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2163.241153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2163.257939] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2164.115895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2164.173472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2164.335601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2164.357296] LNet: 114908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2164.362230] LNet: 114908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2164.367491] LNetError: 114908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2164.372838] LNetError: 114908:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2164.380984] LNet: Removed LNI 192.168.203.10@tcp [ 2164.393588] LNet: Removed LNI 192.168.203.10@tcp2 [ 2164.413158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2164.477307] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2164.483040] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2164.487431] LNet: 114914:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2165.001296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2165.210163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2166.240197] LNet: Removed LNI 192.168.203.10@tcp [ 2166.252725] LNet: Removed LNI 192.168.203.10@tcp2 [ 2166.288572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2166.349803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2166.383092] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2167.079434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2167.102756] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2167.895405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2167.955650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2168.034539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2168.310813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2169.376147] LNet: Removed LNI 192.168.203.10@tcp [ 2170.407066] LNet: Removed LNI 192.168.203.10@tcp2 [ 2170.469095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.547837] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2170.554220] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2170.563798] LNet: 115242:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2171.160076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2171.392280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2171.419194] LNet: Removed LNI 192.168.203.10@tcp [ 2172.456477] LNet: Removed LNI 192.168.203.10@tcp2 [ 2172.506843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2172.599682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2172.623794] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2173.653539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2173.689350] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2174.870620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2174.924503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2174.993828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2175.297649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.316035] LNetError: 115576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2175.321592] LNetError: 115576:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2176.352679] LNet: Removed LNI 192.168.203.10@tcp [ 2176.372227] LNet: Removed LNI 192.168.203.10@tcp2 [ 2176.410556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2176.488203] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2176.497207] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2176.502952] LNet: 115582:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2176.804306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2177.124296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2178.210280] LNet: Removed LNI 192.168.203.10@tcp [ 2178.226181] LNet: Removed LNI 192.168.203.10@tcp2 [ 2178.261222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2178.328269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2178.362522] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2179.149803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2179.175434] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2179.928408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2179.977647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2180.030539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2180.105041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2180.516465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.530460] LNet: 115936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.540979] LNet: 115936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2181.601504] LNet: Removed LNI 192.168.203.10@tcp [ 2182.630633] LNet: Removed LNI 192.168.203.10@tcp2 [ 2182.645416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.696214] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2182.704204] LNet: Added LNI 192.168.203.10@tcp2 [8/256/0/180] [ 2183.189546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2190.100962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2191.715406] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:32:10 (1781238730) [ 2191.906775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2191.942583] LNetError: 116700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2191.953662] LNetError: 116700:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2191.960842] LNet: Removed LNI 192.168.203.10@tcp [ 2193.014809] LNet: Removed LNI 192.168.203.10@tcp2 [ 2193.044718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2193.320639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2193.339921] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2194.055034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2194.079401] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2194.879441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2194.901310] LNet: Removed LNI 192.168.203.10@tcp [ 2194.913482] LNet: Removed LNI 10.1.2.3@tcp [ 2194.931125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.993392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2195.008049] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2195.011513] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2195.407670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2195.429886] LNet: Removed LNI 192.168.203.10@tcp [ 2195.440229] LNet: Removed LNI 10.1.2.3@tcp [ 2200.997393] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:32:20 (1781238740) [ 2201.164527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2201.227519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.305472] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2201.521138] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2201.527851] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2201.536747] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2202.785670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.808822] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2203.515264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2203.545426] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2205.760237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.10@tcp [ 2205.814990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2205.891214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.10@tcp -m PUT -e local_timeout -r 1 [ 2205.952941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2206.023886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2206.027883] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2206.269742] LNetError: 117632:0:(lib-move.c:4710:lnet_parse()) 192.168.203.10@tcp, src 192.168.203.10@tcp: Dropping PUT (error -108 looking up sender) [ 2208.288139] LNet: Removed LNI 192.168.203.10@tcp [ 2208.303137] LNet: Removed LNI 10.1.2.3@tcp [ 2208.752664] Key type .llcrypt unregistered [ 2208.757259] Key type ._llcrypt unregistered [ 2219.832723] Key type ._llcrypt registered [ 2219.836190] Key type .llcrypt registered [ 2221.106330] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:32:39 (1781238759) [ 2221.553992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2221.568231] alg: No test for adler32 (adler32-zlib) [ 2222.488860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.891730] LNet: 118943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2223.466349] Key type .llcrypt unregistered [ 2223.468306] Key type ._llcrypt unregistered [ 2229.391205] Key type ._llcrypt registered [ 2229.394269] Key type .llcrypt registered [ 2229.654596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2229.669935] alg: No test for adler32 (adler32-zlib) [ 2230.630524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2230.671744] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2237.864424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 2238.070163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2238.688433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.110@tcp [ 2238.881503] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.110@tcp [ 2238.927680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2238.955152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2238.982279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2239.008305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2239.039241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2239.076574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2239.103973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2239.151216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2239.170848] LNet: 119392:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2239.187519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2239.217223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2239.247171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2239.289182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2239.312529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2239.341164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2239.957987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2240.167176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2240.201398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2240.230747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2240.303769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2240.359904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2240.391129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2240.435223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2240.507187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2240.550326] LNet: 119510:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2240.600761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2240.647634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2240.688802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2240.734972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2240.796594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2240.840316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2241.548799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.110@tcp [ 2241.832399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2241.873603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2241.911031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2241.951516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.012987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.053221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.084610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.122684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.162960] LNet: 119627:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.193794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2242.230659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2242.300843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2242.356261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2242.403406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2242.432815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2243.099482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2243.379861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2243.411194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2243.448816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2243.471059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2243.494317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2243.526361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2243.569517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2243.596599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2243.639550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2243.673722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2243.738889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2243.771919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2243.801503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2243.838381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2244.191333] LNet: 119790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.198837] LNetError: 119790:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.216370] LNet: Removed LNI 192.168.203.10@tcp [ 2244.573918] Key type .llcrypt unregistered [ 2244.577956] Key type ._llcrypt unregistered [ 2260.705926] Key type ._llcrypt registered [ 2260.709330] Key type .llcrypt registered [ 2262.182556] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:33:20 (1781238800) [ 2262.585880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2262.617285] alg: No test for adler32 (adler32-zlib) [ 2263.541896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.912553] LNet: 120677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.452855] Key type .llcrypt unregistered [ 2264.455725] Key type ._llcrypt unregistered [ 2269.950627] Key type ._llcrypt registered [ 2269.956430] Key type .llcrypt registered [ 2270.266481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2270.285614] alg: No test for adler32 (adler32-zlib) [ 2271.244114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2271.293795] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2278.416568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.110@tcp [ 2286.578345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2286.604898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2286.623460] LNet: Added LNI 192.168.203.10@tcp1 [8/256/0/180] [ 2287.350485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2287.915197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.110@tcp [ 2288.152680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2288.180850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2288.205076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2288.231179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2288.253230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2288.277878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2288.305491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2288.336642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2288.352675] LNet: 121329:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2288.370287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2288.400888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2288.424831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2288.455434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2288.495519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2288.528114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2289.206610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2289.398304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2289.429292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2289.460630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2289.490687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2289.524415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2289.557504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2289.593165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2289.636597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2289.663759] LNet: 121446:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2289.695861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2289.724242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2289.759993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2289.760149] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.110@tcp, 192.168.203.110@tcp1 [ 2289.800142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2289.840786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2289.907833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2290.602344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.110@tcp [ 2290.825987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2290.864326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2290.897314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2290.934384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2290.965719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2291.003778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2291.037644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2291.069893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2291.084707] LNet: 121563:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2291.104333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2291.128749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2291.157545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2291.205264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2291.253636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2291.301964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2292.016775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2292.246165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2292.281984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2292.323580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2292.356381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2292.391995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2292.416152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2292.436330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2292.459391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2292.488611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2292.531584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2292.579584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2292.614272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2292.648368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2292.671686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2297.321877] LNet: 121761:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2297.331819] LNetError: 121761:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2297.346074] LNet: Removed LNI 192.168.203.10@tcp [ 2297.367625] LNet: Removed LNI 192.168.203.10@tcp1 [ 2297.974946] Key type .llcrypt unregistered [ 2297.978252] Key type ._llcrypt unregistered [ 2313.558976] Key type ._llcrypt registered [ 2313.561174] Key type .llcrypt registered [ 2314.949452] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:34:13 (1781238853) [ 2321.533407] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:34:20 (1781238860) [ 2328.425339] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:34:27 (1781238867) [ 2335.805269] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:34:34 (1781238874) [ 2344.065714] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:34:42 (1781238882) [ 2353.684196] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:34:51 (1781238891) [ 2362.043842] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:35:00 (1781238900) [ 2369.674702] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:35:08 (1781238908) [ 2378.217595] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:35:16 (1781238916) [ 2385.434700] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:35:24 (1781238924) [ 2392.963219] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:35:31 (1781238931) [ 2400.269729] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:35:38 (1781238938) [ 2408.249662] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:35:46 (1781238946) [ 2415.371265] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:35:54 (1781238954) [ 2422.646960] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:36:01 (1781238961) [ 2429.580722] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:36:08 (1781238968) [ 2437.746129] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:36:16 (1781238976) [ 2444.345152] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:36:23 (1781238983) [ 2444.800912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2444.819697] alg: No test for adler32 (adler32-zlib) [ 2445.721269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.808747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.825304] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2446.664971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2446.680367] LNet: 132472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2446.687379] LNetError: 132472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2446.705642] LNet: Removed LNI 192.168.203.10@tcp [ 2446.723664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.766326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2446.782644] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2452.088685] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:36:31 (1781238991) [ 2458.750231] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:36:37 (1781238997) [ 2465.569478] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:36:44 (1781239004) [ 2472.552505] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:36:51 (1781239011) [ 2478.317720] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:36:57 (1781239017) [ 2485.365227] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:37:04 (1781239024) [ 2492.133596] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:37:10 (1781239030) [ 2499.396331] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:37:18 (1781239038) [ 2505.693052] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:37:24 (1781239044) [ 2512.544185] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:37:30 (1781239050) [ 2519.894868] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:37:38 (1781239058) [ 2526.497166] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:37:45 (1781239065) [ 2533.617364] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2285 sec ========== 00:37:52 (1781239072) [ 2534.077138] LNet: 139938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2534.091506] LNetError: 139938:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2534.111321] LNet: Removed LNI 192.168.203.10@tcp [ 2534.828552] Key type .llcrypt unregistered [ 2534.835978] Key type ._llcrypt unregistered [ 2547.128268] Key type ._llcrypt registered [ 2547.134032] Key type .llcrypt registered [ 2547.775160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2547.803176] alg: No test for adler32 (adler32-zlib) [ 2548.955428] Lustre: Lustre: Build Version: 2.17.53_79_gff07dcf [ 2549.273692] LNet: Added LNI 192.168.203.10@tcp [8/256/0/180] [ 2550.968189] Key type lgssc registered [ 2552.331989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2604.468838] Lustre: Mounted lustre-client [ 2608.866777] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2615.789061] Lustre: Unmounted lustre-client [ 2641.892808] Key type lgssc unregistered [ 2642.167769] LNet: 142867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2642.175170] LNetError: 142867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2642.206943] LNet: Removed LNI 192.168.203.10@tcp [ 2642.886389] Key type .llcrypt unregistered [ 2642.889951] Key type ._llcrypt unregistered