[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 447002468 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002377] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003021] kvm-guest: setup PV IPIs [ 0.005611] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007014] pid_max: default: 32768 minimum: 301 [ 0.009083] LSM: Security Framework initializing [ 0.010064] Yama: becoming mindful. [ 0.011051] SELinux: Initializing. [ 0.012086] *** VALIDATE selinux *** [ 0.020081] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025927] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027054] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028130] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029178] *** VALIDATE tmpfs *** [ 0.031441] *** VALIDATE proc *** [ 0.032096] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034238] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035165] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037031] Spectre V2 : User space: Vulnerable [ 0.038010] Speculative Store Bypass: Vulnerable [ 0.041186] debug: unmapping init [mem 0xffffffff90059000-0xffffffff90060fff] [ 0.043159] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044677] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046015] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048015] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051012] ... event mask: 000000070000000f [ 0.052305] rcu: Hierarchical SRCU implementation. [ 0.054469] smp: Bringing up secondary CPUs ... [ 0.055771] x86: Booting SMP configuration: [ 0.056027] .... node #0, CPUs: #1 #2 #3 [ 0.060030] smp: Brought up 1 node, 4 CPUs [ 0.062028] smpboot: Max logical packages: 1 [ 0.063018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135465] node 0 deferred pages initialised in 70ms [ 0.139110] devtmpfs: initialized [ 0.140274] x86/mm: Memory block size: 128MB [ 0.142000] gcov: version magic: 0x41383552 [ 0.144303] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.150307] pinctrl core: initialized pinctrl subsystem [ 0.152194] [ 0.152667] ************************************************************* [ 0.154011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157023] ** ** [ 0.159010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161013] ** ** [ 0.163014] ** This means that this kernel is built to expose internal ** [ 0.165011] ** IOMMU data structures, which may compromise security on ** [ 0.168013] ** your system. ** [ 0.170012] ** ** [ 0.172014] ** If you see this message and you are not debugging the ** [ 0.175013] ** kernel, report this immediately to your vendor! ** [ 0.177012] ** ** [ 0.179015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182015] ************************************************************* [ 0.183670] NET: Registered protocol family 16 [ 0.185416] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.188062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.189064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.192096] cpuidle: using governor menu [ 0.193571] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.196463] PCI: Using configuration type 1 for base access [ 0.198142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.207413] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211046] cryptd: max_cpu_qlen set to 1000 [ 0.213266] ACPI: Added _OSI(Module Device) [ 0.214000] ACPI: Added _OSI(Processor Device) [ 0.214000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219396] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.225505] ACPI: Interpreter enabled [ 0.227070] ACPI: PM: (supports S0 S3 S4 S5) [ 0.228000] ACPI: Using IOAPIC for interrupt routing [ 0.228000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231390] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.241503] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.244034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.246023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.249074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254254] acpiphp: Slot [2] registered [ 0.256113] acpiphp: Slot [3] registered [ 0.257129] acpiphp: Slot [4] registered [ 0.259106] acpiphp: Slot [5] registered [ 0.261153] acpiphp: Slot [6] registered [ 0.263135] acpiphp: Slot [7] registered [ 0.265118] acpiphp: Slot [8] registered [ 0.267132] acpiphp: Slot [9] registered [ 0.269106] acpiphp: Slot [10] registered [ 0.270467] acpiphp: Slot [11] registered [ 0.273124] acpiphp: Slot [12] registered [ 0.274105] acpiphp: Slot [13] registered [ 0.276101] acpiphp: Slot [14] registered [ 0.279125] acpiphp: Slot [15] registered [ 0.280138] acpiphp: Slot [16] registered [ 0.283123] acpiphp: Slot [17] registered [ 0.285756] acpiphp: Slot [18] registered [ 0.287110] acpiphp: Slot [19] registered [ 0.288436] acpiphp: Slot [20] registered [ 0.290122] acpiphp: Slot [21] registered [ 0.292110] acpiphp: Slot [22] registered [ 0.294111] acpiphp: Slot [23] registered [ 0.295095] acpiphp: Slot [24] registered [ 0.298159] acpiphp: Slot [25] registered [ 0.300096] acpiphp: Slot [26] registered [ 0.302147] acpiphp: Slot [27] registered [ 0.303111] acpiphp: Slot [28] registered [ 0.305118] acpiphp: Slot [29] registered [ 0.306115] acpiphp: Slot [30] registered [ 0.308117] acpiphp: Slot [31] registered [ 0.310070] PCI host bridge to bus 0000:00 [ 0.312021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.314024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.316023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.319026] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.321019] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.324032] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.326185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.330159] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.336000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.344015] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.348060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.352021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.354018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357027] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.359313] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.361995] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.365046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.370068] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.375014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.387015] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.392016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.397000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.401020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.405016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417027] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.425000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.433016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.440016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.451017] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.463483] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.465402] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.469420] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.473349] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.475236] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.482071] iommu: Default domain type: Passthrough [ 0.484468] SCSI subsystem initialized [ 0.485000] ACPI: bus type USB registered [ 0.485000] usbcore: registered new interface driver usbfs [ 0.487102] usbcore: registered new interface driver hub [ 0.490098] usbcore: registered new device driver usb [ 0.492164] pps_core: LinuxPPS API ver. 1 registered [ 0.494021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.498051] PTP clock support registered [ 0.501118] EDAC MC: Ver: 3.0.0 [ 0.503412] PCI: Using ACPI for IRQ routing [ 0.505515] NetLabel: Initializing [ 0.507016] NetLabel: domain hash size = 128 [ 0.510014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.512100] NetLabel: unlabeled traffic allowed by default [ 0.514115] vgaarb: loaded [ 0.516289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.517012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.523375] clocksource: Switched to clocksource kvm-clock [ 0.662837] VFS: Disk quotas dquot_6.6.0 [ 0.664591] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.668722] *** VALIDATE ramfs *** [ 0.669903] *** VALIDATE hugetlbfs *** [ 0.672228] pnp: PnP ACPI init [ 0.676304] pnp: PnP ACPI: found 6 devices [ 0.696985] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.700323] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.702295] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.704568] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.706377] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.708604] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.711040] NET: Registered protocol family 2 [ 0.714292] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.719055] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.724258] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.728916] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.732394] TCP: Hash tables configured (established 65536 bind 65536) [ 0.736102] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.738968] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.741680] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.744534] NET: Registered protocol family 1 [ 0.746923] RPC: Registered named UNIX socket transport module. [ 0.749335] RPC: Registered udp transport module. [ 0.751186] RPC: Registered tcp transport module. [ 0.752701] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.754835] NET: Registered protocol family 44 [ 0.756309] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.758427] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.760973] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.763395] PCI: CLS 0 bytes, default 64 [ 0.764931] Unpacking initramfs... [ 2.147431] debug: unmapping init [mem 0xffff9d67fcc64000-0xffff9d67fffcffff] [ 2.151319] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.153483] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.156275] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.686143] Initialise system trusted keyrings [ 2.688881] Key type blacklist registered [ 2.690893] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.699476] zbud: loaded [ 2.702547] *** VALIDATE nfs *** [ 2.704114] *** VALIDATE nfs4 *** [ 2.705660] pstore: using deflate compression [ 2.708660] Platform Keyring initialized [ 2.818074] NET: Registered protocol family 38 [ 2.819511] Key type asymmetric registered [ 2.821013] Asymmetric key parser 'x509' registered [ 2.825598] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.828348] io scheduler mq-deadline registered [ 2.830067] io scheduler kyber registered [ 2.831873] io scheduler bfq registered [ 2.834471] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.837379] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.840588] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.843622] ACPI: Power Button [PWRF] [ 2.936729] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.031730] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.150437] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.183343] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.216503] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.223826] Non-volatile memory driver v1.3 [ 3.225128] Linux agpgart interface v0.103 [ 3.257030] virtio_blk virtio1: [vda] 133640 512-byte logical blocks (68.4 MB/65.3 MiB) [ 3.259636] vda: detected capacity change from 0 to 68423680 [ 3.273898] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.276555] vdb: detected capacity change from 0 to 1073741824 [ 3.284208] libphy: Fixed MDIO Bus: probed [ 3.293179] usbcore: registered new interface driver usbserial_generic [ 3.295635] usbserial: USB Serial support registered for generic [ 3.297784] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.300783] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302583] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.305087] mousedev: PS/2 mouse device common for all mice [ 3.307947] rtc_cmos 00:05: RTC can wake from S4 [ 3.309817] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.311646] rtc_cmos 00:05: registered as rtc0 [ 3.315843] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.317566] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.321136] intel_pstate: CPU model not supported [ 3.324659] hid: raw HID events driver (C) Jiri Kosina [ 3.332680] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.334031] usbcore: registered new interface driver usbhid [ 3.339436] usbhid: USB HID core driver [ 3.341635] drop_monitor: Initializing network drop monitor service [ 3.344446] Initializing XFRM netlink socket [ 3.346468] NET: Registered protocol family 10 [ 3.349635] Segment Routing with IPv6 [ 3.351052] NET: Registered protocol family 17 [ 3.353395] mpls_gso: MPLS GSO support [ 3.361120] RAS: Correctable Errors collector initialized. [ 3.362653] AVX version of gcm_enc/dec engaged. [ 3.363642] AES CTR mode by8 optimization enabled [ 3.449808] sched_clock: Marking stable (3449775188, 0)->(4408553870, -958778682) [ 3.453645] registered taskstats version 1 [ 3.456150] Loading compiled-in X.509 certificates [ 3.461142] zswap: loaded using pool lzo/zbud [ 3.488538] Key type big_key registered [ 3.507178] Key type encrypted registered [ 3.509037] ima: No TPM chip found, activating TPM-bypass! [ 3.511281] ima: Allocated hash algorithm: sha1 [ 3.513253] ima: No architecture policies found [ 3.514600] evm: Initialising EVM extended attributes: [ 3.516076] evm: security.selinux [ 3.517253] evm: security.ima [ 3.518376] evm: security.capability [ 3.519680] evm: HMAC attrs: 0x1 [ 3.522362] rtc_cmos 00:05: setting system clock to 2025-11-14 20:58:49 UTC (1763153929) [ 3.529062] debug: unmapping init [mem 0xffffffff91003000-0xffffffff911fffff] [ 3.531971] debug: unmapping init [mem 0xffffffff8fd82000-0xffffffff90058fff] [ 3.541088] Write protecting the kernel read-only data: 28672k [ 3.543717] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.546622] debug: unmapping init [mem 0xffffffff8ed14000-0xffffffff8edfffff] [ 3.583138] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.591218] systemd[1]: Detected virtualization kvm. [ 3.593169] systemd[1]: Detected architecture x86-64. [ 3.594759] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.617363] systemd[1]: No hostname configured. [ 3.619090] systemd[1]: Set hostname to . [ 3.621348] random: systemd: uninitialized urandom read (16 bytes read) [ 3.623832] systemd[1]: Initializing machine ID from random generator. [ 3.680517] random: ln: uninitialized urandom read (6 bytes read) [ 3.777742] random: systemd: uninitialized urandom read (16 bytes read) [ 3.780289] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.784293] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.787982] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.454961] device-mapper: uevent: version 1.0.3 [ 4.457345] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.260571] virtio_net virtio0 ens2: renamed from eth0 [ 5.264842] scsi host0: ata_piix [ 5.270337] random: fast init done [ 5.280491] scsi host1: ata_piix [ 5.282626] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.285158] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.660315] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.025601] random: crng init done [ 10.026811] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.447578] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.680336] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.958275] SELinux: Disabled at runtime. [ 12.019595] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.025831] systemd[1]: Detected virtualization kvm. [ 12.027521] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.575379] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.579252] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.586896] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.590955] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.594466] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.602394] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.609995] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ 12.798473] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 13.071993] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.460421] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.470958] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.569434] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.593619] EDAC sbridge: Ver: 1.1.2 [ 14.898287] Key type dns_resolver registered [ 15.205963] NFS: Registering the id_resolver key type [ 15.207879] Key type id_resolver registered [ 15.209305] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg329-client login: [ 45.177805] libcfs: loading out-of-tree module taints kernel. [ 45.225268] Key type ._llcrypt registered [ 45.226895] Key type .llcrypt registered [ 45.498040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.505222] alg: No test for adler32 (adler32-zlib) [ 46.555468] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 46.901538] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 48.536177] Key type lgssc registered [ 49.250845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.260012] hrtimer: interrupt took 4003772 ns [ 147.248447] Lustre: Mounted lustre-client [ 151.774888] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 167.342735] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing check_logdir /tmp/testlogs/ [ 170.374199] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing yml_node [ 173.026336] Lustre: lustre-OST0000-osc-ffff9d6849629800: disconnect after 24s idle [ 174.005777] Lustre: DEBUG MARKER: Client: 2.16.61.2 [ 176.144657] Lustre: DEBUG MARKER: MDS: 2.16.61.2 [ 178.273418] Lustre: DEBUG MARKER: OSS: 2.16.61.2 [ 179.507227] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Nov 14 16:01:44 EST 2025 [ 191.334794] LustreError: 7908:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d6849629800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 191.348802] LustreError: 7908:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 191.393523] Lustre: Unmounted lustre-client [ 224.074850] Key type lgssc unregistered [ 224.311901] LNet: 8356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 224.322871] LNetError: 8356:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 224.344133] LNet: Removed LNI 192.168.203.29@tcp [ 225.051170] Key type .llcrypt unregistered [ 225.055536] Key type ._llcrypt unregistered [ 245.998979] Key type ._llcrypt registered [ 246.001061] Key type .llcrypt registered [ 246.156840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 246.169959] alg: No test for adler32 (adler32-zlib) [ 247.076688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 247.154933] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 247.180554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 250.621402] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 250.886407] LNet: 9229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 250.902580] LNetError: 9229:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 250.925751] LNet: Removed LNI 192.168.203.29@tcp [ 251.453269] Key type .llcrypt unregistered [ 251.455303] Key type ._llcrypt unregistered [ 255.997606] Key type ._llcrypt registered [ 255.999792] Key type .llcrypt registered [ 257.195546] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 16:03:01 (1763154181) [ 257.468905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 257.483971] alg: No test for adler32 (adler32-zlib) [ 258.466635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.549619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.573128] LNet: 9634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.605822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.651459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 258.670925] LNet: 9641:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 264.975867] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 16:03:09 (1763154189) [ 265.079213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.106360] LNet: 10215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 265.134556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.217658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 270.143861] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 16:03:15 (1763154195) [ 270.249204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.276987] LNet: 10796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.297350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.404206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 275.581780] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 16:03:20 (1763154200) [ 275.649248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.715341] LNet: 11380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.747236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.849036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 281.841461] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 16:03:26 (1763154206) [ 282.297954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.330566] LNet: 11972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 282.369719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.557923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 283.313656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 290.201194] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 16:03:35 (1763154215) [ 290.424158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 290.442596] LNet: 12562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 290.459828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 290.529485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 290.784441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 291.200405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 298.097733] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 16:03:42 (1763154222) [ 298.187162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 298.229518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 298.321498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 303.610228] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 16:03:48 (1763154228) [ 303.862066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 303.915591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 303.996651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 304.033340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 304.399217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 304.429415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 304.798899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 304.834868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 305.359498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 305.434275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 305.937900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 305.979119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 306.475614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 306.535687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 306.921535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 306.968582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 307.592613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 307.631501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 308.019960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 308.051184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 308.441598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 308.473229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 308.877394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 308.927284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 309.359569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 309.398608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 309.763389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 309.822852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 315.011490] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 16:03:59 (1763154239) [ 315.282392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.310089] LNet: 14424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.320426] LNet: 14424:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 315.340380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.404156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 315.459292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 320.719173] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 16:04:05 (1763154245) [ 320.855219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.886940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.966119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 321.002199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 326.758314] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 16:04:11 (1763154251) [ 326.965924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.028562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.087724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 327.110595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 331.929455] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 16:04:17 (1763154257) [ 332.074051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.101808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.158545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 332.181208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 337.336854] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 16:04:22 (1763154262) [ 337.509946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.557543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.622839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 337.657727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 343.916949] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 16:04:28 (1763154268) [ 344.255674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.305199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.400236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 344.431749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 350.565487] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 16:04:35 (1763154275) [ 350.797357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.821880] LNet: 17944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.827422] LNet: 17944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 350.850055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.979383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 351.347639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 357.085156] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 16:04:41 (1763154281) [ 357.133793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 357.169132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 357.372860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 357.599811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 362.770491] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 16:04:47 (1763154287) [ 362.870436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 362.903378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.003293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 363.164618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 367.739072] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 16:04:52 (1763154292) [ 367.869613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.928371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.298896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 368.512363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 373.840261] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 16:04:58 (1763154298) [ 374.051836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.077729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.164631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 374.198753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 379.548826] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 16:05:04 (1763154304) [ 379.708266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 379.763075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.831384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.886804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 386.070206] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 16:05:10 (1763154310) [ 386.237239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.282841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.327802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 392.095727] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 16:05:17 (1763154317) [ 392.424907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.462568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.506211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 397.559021] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 16:05:22 (1763154322) [ 397.725662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.752976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.805656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 398.094455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 402.691465] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 16:05:27 (1763154327) [ 402.774814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.809730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.878422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 402.908121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 407.198063] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 16:05:32 (1763154332) [ 407.349384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.368442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.410913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 407.445642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 411.669554] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 16:05:36 (1763154336) [ 411.761571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.795774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.851717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 411.875280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 415.953717] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 16:05:41 (1763154341) [ 416.017356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.044601] LNet: 25133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.048807] LNet: 25133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 416.072180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.134204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 416.159692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 420.330631] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 16:05:45 (1763154345) [ 420.574479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.604297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.646419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 420.705717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 420.724307] LNetError: 25740:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 420.756590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 420.797447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 425.059775] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 16:05:50 (1763154350) [ 425.164789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.188607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.254798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YtZwavzdKY [ 425.303889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YtZwavzdKY [ 425.344887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.YtZwavzdKY [ 429.814856] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 16:05:54 (1763154354) [ 429.922960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 429.976434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.060634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 430.089776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 430.136776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 434.450360] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 16:05:59 (1763154359) [ 434.639116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.669580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.714892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 439.749066] LNet: Message forwarding will be enabled in 2s [ 443.872125] LNet: Message forwarding enabled [ 444.228866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 444.260397] LNet: Message forwarding will stop in 2s to 6s [ 448.587462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 449.015286] LNet: Message forwarding disabled [ 450.239352] LNet: Message forwarding will be enabled in 2s [ 454.176185] LNet: Message forwarding enabled [ 454.811959] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 16:06:19 (1763154379) [ 454.908916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.882950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.939249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 460.666309] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 16:06:25 (1763154385) [ 460.744495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.799576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.855851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 460.885540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 460.911685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 460.937378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 460.963542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 460.986551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 461.006503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 461.043147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 461.077858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 461.123833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 461.162194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 461.203495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 461.231561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 461.264596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 461.290731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 461.321380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 461.345099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 461.374492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 461.406937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 461.426119] LNetError: 28818:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 466.036181] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 16:06:31 (1763154391) [ 466.097513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.130153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.176847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.870735] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 16:06:35 (1763154395) [ 472.183162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg329-server [ 473.579493] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 16:06:38 (1763154398) [ 473.653299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.679200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.790661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 473.811335] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 475.202361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[30-34]@tcp [ 475.216650] LNet: 30533:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 475.266164] LNetError: 30535:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.33@tcp has gone from up to down [ 475.297564] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 476.262529] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.31@tcp is being used as a gateway but routing feature is not turned on [ 476.278142] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 477.283948] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.31@tcp is being used as a gateway but routing feature is not turned on [ 477.306428] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 479.404764] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 479.420407] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 480.646266] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 16:06:45 (1763154405) [ 482.177317] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg329-server [ 483.493575] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.31@tcp is being used as a gateway but routing feature is not turned on [ 483.500820] LNetError: 30323:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 15 previous similar messages [ 483.661597] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 16:06:48 (1763154408) [ 483.898134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.935836] LNetError: 31468:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 483.959638] LNet: Removed LNI 192.168.203.29@tcp [ 483.981547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.028367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 484.050059] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 484.622290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[30-35/2]@tcp [ 484.655858] LNet: 31586:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 484.660804] LNet: 31586:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 484.682901] LNetError: 31472:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.32@tcp has gone from up to down [ 484.684957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[30-35/2]@tcp [ 484.695494] LNetError: 31472:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 490.181285] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 16:06:54 (1763154414) [ 490.475045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.494024] LNetError: 32179:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 490.515367] LNet: Removed LNI 192.168.203.29@tcp [ 490.540898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.646464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 490.716471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.748844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.818605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.879644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.908614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.969535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 491.060867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.119621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.241603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.310651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.362839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.420289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 491.501191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.555206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.615992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.694313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.726466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.773488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 491.832031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.867165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.925387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 491.980825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.013920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.079499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 492.173122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.201214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.252095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 492.309734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.343994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.395165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 492.450354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.473797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.535167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 498.232661] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 16:07:02 (1763154422) [ 500.169140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg329-server [ 501.808144] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 16:07:06 (1763154426) [ 503.295627] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg329-server [ 505.485114] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 16:07:09 (1763154429) [ 506.589050] Key type .llcrypt unregistered [ 506.595026] Key type ._llcrypt unregistered [ 511.366861] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 511.482050] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 511.487884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 511.493686] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 513.987035] Key type ._llcrypt registered [ 513.988747] Key type .llcrypt registered [ 514.136738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.150108] alg: No test for adler32 (adler32-zlib) [ 515.034368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.127573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 515.157906] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 515.710175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 515.725883] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 517.421498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 517.446600] LNet: Removed LNI 10.1.2.3@tcp [ 517.557522] LNet: 34293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 517.576714] LNetError: 34293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 517.587318] LNet: Removed LNI 192.168.203.29@tcp [ 518.105442] Key type .llcrypt unregistered [ 518.107259] Key type ._llcrypt unregistered [ 526.971548] Key type ._llcrypt registered [ 526.972784] Key type .llcrypt registered [ 528.021126] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 16:07:32 (1763154452) [ 528.561554] Key type .llcrypt unregistered [ 528.564504] Key type ._llcrypt unregistered [ 533.202383] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 533.306856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 533.309719] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 533.315926] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 535.939842] Key type ._llcrypt registered [ 535.941364] Key type .llcrypt registered [ 536.117804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 536.128996] alg: No test for adler32 (adler32-zlib) [ 537.028165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.089978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 537.116562] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 537.833552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 537.868600] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 540.502352] LNet: 35772:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.518569] LNetError: 35772:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 540.527141] LNet: Removed LNI 192.168.203.29@tcp [ 540.542560] LNet: Removed LNI 10.1.2.3@tcp [ 541.199448] Key type .llcrypt unregistered [ 541.201594] Key type ._llcrypt unregistered [ 551.033553] Key type ._llcrypt registered [ 551.037456] Key type .llcrypt registered [ 552.317524] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 16:07:56 (1763154476) [ 552.869343] Key type .llcrypt unregistered [ 552.870613] Key type ._llcrypt unregistered [ 558.920430] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 559.145025] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 559.148852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 559.151748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 561.786911] Key type ._llcrypt registered [ 561.788546] Key type .llcrypt registered [ 562.076545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 562.106762] alg: No test for adler32 (adler32-zlib) [ 563.055519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.122886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 563.164461] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 565.511076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 565.537511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 567.632238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 567.674465] LNet: Removed LNI 10.1.2.3@tcp [ 567.690632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 567.720922] LNet: Removed LNI 10.1.2.31@tcp [ 567.742304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 567.770949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 568.595655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 568.625215] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 569.592635] LNet: 37681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.597707] LNetError: 37681:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 569.600526] LNet: Removed LNI 10.1.2.3@tcp [ 569.611302] LNet: Removed LNI 10.1.2.31@tcp [ 570.446502] Key type .llcrypt unregistered [ 570.449198] Key type ._llcrypt unregistered [ 581.543115] Key type ._llcrypt registered [ 581.544783] Key type .llcrypt registered [ 582.783356] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 16:08:27 (1763154507) [ 583.374685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 583.390574] alg: No test for adler32 (adler32-zlib) [ 584.278376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.318543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 584.350889] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 585.060835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 585.078638] LNet: 38630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 585.082894] LNetError: 38630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 585.096806] LNet: Removed LNI 192.168.203.29@tcp [ 585.119412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.168303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.29@tcp [ 585.187151] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 585.883072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.29@tcp [ 585.921127] LNet: Removed LNI 192.168.203.29@tcp [ 585.993305] LNet: 38765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.485431] Key type .llcrypt unregistered [ 586.491268] Key type ._llcrypt unregistered [ 596.796486] Key type ._llcrypt registered [ 596.797945] Key type .llcrypt registered [ 598.269082] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 16:08:42 (1763154522) [ 600.209115] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg329-server [ 602.569802] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 16:08:46 (1763154526) [ 603.167231] Key type .llcrypt unregistered [ 603.172406] Key type ._llcrypt unregistered [ 608.566554] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 608.632976] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 608.640100] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 608.643056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 611.368751] Key type ._llcrypt registered [ 611.370442] Key type .llcrypt registered [ 611.635553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 611.663190] alg: No test for adler32 (adler32-zlib) [ 612.618512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.677090] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 613.596096] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 616.098179] LNet: 40595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.107739] LNetError: 40595:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.113394] LNet: Removed LNI 192.168.203.29@tcp1 [ 616.141382] LNet: Removed LNI 10.1.2.3@tcp2 [ 616.741391] Key type .llcrypt unregistered [ 616.749498] Key type ._llcrypt unregistered [ 628.425986] Key type ._llcrypt registered [ 628.427498] Key type .llcrypt registered [ 629.555754] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 16:09:14 (1763154554) [ 630.175975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 630.197749] alg: No test for adler32 (adler32-zlib) [ 631.162069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.243615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 631.276976] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 632.203603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 632.236121] LNet: 41560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.250037] LNetError: 41560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.268205] LNet: Removed LNI 192.168.203.29@tcp [ 632.292947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 632.385170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 632.424553] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 638.432720] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 16:09:23 (1763154563) [ 638.609707] LNet: 42152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.631151] LNetError: 42152:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.657360] LNet: Removed LNI 192.168.203.29@tcp [ 639.145383] Key type .llcrypt unregistered [ 639.147050] Key type ._llcrypt unregistered [ 643.658064] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 643.853693] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 643.883296] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 643.902611] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 646.425941] Key type ._llcrypt registered [ 646.427302] Key type .llcrypt registered [ 646.642227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 646.661305] alg: No test for adler32 (adler32-zlib) [ 647.572490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.627696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.664446] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 648.307087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 648.321844] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.100900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.130232] LNet: 42890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.136577] LNetError: 42890:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.143947] LNet: Removed LNI 192.168.203.29@tcp [ 650.150697] LNet: Removed LNI 10.1.2.3@tcp [ 650.169963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.227530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 650.262306] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 650.267573] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 650.652503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.670172] LNet: 42917:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.679702] LNetError: 42917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 650.696181] LNetError: 42917:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 650.700978] LNet: Removed LNI 192.168.203.29@tcp [ 650.718613] LNet: Removed LNI 10.1.2.3@tcp [ 650.751038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.827743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 650.848898] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 650.853576] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 651.309791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 651.326050] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 652.210487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.245586] LNet: 43079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.251413] LNetError: 43079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.254740] LNetError: 43079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 652.260697] LNet: Removed LNI 192.168.203.29@tcp [ 652.263404] LNet: Removed LNI 10.1.2.3@tcp [ 652.271931] LNet: Removed LNI 192.168.203.29@tcp2 [ 652.293704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.382368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 652.413321] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 652.419736] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 652.426352] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 658.296880] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 16:09:43 (1763154583) [ 658.516517] LNet: 43677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.531761] LNetError: 43677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.534947] LNetError: 43677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 658.548178] LNet: Removed LNI 192.168.203.29@tcp [ 658.552814] LNet: Removed LNI 10.1.2.3@tcp [ 658.560688] LNet: Removed LNI 192.168.203.29@tcp2 [ 659.139336] Key type .llcrypt unregistered [ 659.141075] Key type ._llcrypt unregistered [ 663.887635] Key type ._llcrypt registered [ 663.889069] Key type .llcrypt registered [ 664.101207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.127677] alg: No test for adler32 (adler32-zlib) [ 665.042951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.125641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 665.152968] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 665.839189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.857776] LNet: 44068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.866254] LNetError: 44068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.877746] LNet: Removed LNI 192.168.203.29@tcp [ 665.899772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.959695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 665.986430] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 666.436337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.463970] LNet: 44098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.474580] LNetError: 44098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 666.489606] LNet: Removed LNI 192.168.203.29@tcp [ 666.520235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.613788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 666.644711] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 667.109622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 667.149967] LNet: Removed LNI 192.168.203.29@tcp [ 667.166913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.268033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 667.319502] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 667.815797] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 667.979705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 667.985299] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 667.987613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.364766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.397808] LNet: 44206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.407699] LNet: 44206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 670.423250] LNetError: 44206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.439019] LNetError: 44206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 670.468209] LNet: Removed LNI 192.168.203.29@tcp [ 670.517796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.616574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 670.667933] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 676.429780] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 16:10:01 (1763154601) [ 676.569313] LNet: 44797:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 676.581559] LNetError: 44797:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 676.633263] LNet: Removed LNI 192.168.203.29@tcp [ 677.337969] Key type .llcrypt unregistered [ 677.339745] Key type ._llcrypt unregistered [ 682.846234] Key type ._llcrypt registered [ 682.848039] Key type .llcrypt registered [ 683.049276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 683.071035] alg: No test for adler32 (adler32-zlib) [ 684.046968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.121853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 684.169533] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 685.011529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.038849] LNet: 45198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.044197] LNetError: 45198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.063020] LNet: Removed LNI 192.168.203.29@tcp [ 685.077923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.131469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 685.156840] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 691.584972] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 16:10:16 (1763154616) [ 691.677977] LNet: 45784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.686140] LNetError: 45784:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.715113] LNet: Removed LNI 192.168.203.29@tcp [ 692.203403] Key type .llcrypt unregistered [ 692.207286] Key type ._llcrypt unregistered [ 696.660980] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 696.783272] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 696.788684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 696.801304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.251222] Key type ._llcrypt registered [ 699.252960] Key type .llcrypt registered [ 699.454194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.471943] alg: No test for adler32 (adler32-zlib) [ 700.386150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.446377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.473184] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 701.294921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 701.303800] LNet: 46236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.310159] LNetError: 46236:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.317702] LNet: Removed LNI 192.168.203.29@tcp [ 701.356309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 701.459273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 701.508183] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 707.155827] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 16:10:32 (1763154632) [ 707.436770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.466710] LNet: 46834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.481905] LNetError: 46834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 707.506530] LNet: Removed LNI 192.168.203.29@tcp [ 707.523716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.626535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 713.186903] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 16:10:37 (1763154637) [ 713.270083] LNet: 47408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.959605] Key type .llcrypt unregistered [ 713.963866] Key type ._llcrypt unregistered [ 719.210093] Key type ._llcrypt registered [ 719.211763] Key type .llcrypt registered [ 719.432765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 719.445937] alg: No test for adler32 (adler32-zlib) [ 720.331565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.405725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.434413] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 721.243223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.260045] LNet: 47809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.269823] LNetError: 47809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.284711] LNet: Removed LNI 192.168.203.29@tcp [ 721.319880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.373855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 721.398880] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 727.104941] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 16:10:51 (1763154651) [ 727.379648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.406950] LNet: 48403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.422143] LNetError: 48403:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.434722] LNet: Removed LNI 192.168.203.29@tcp [ 727.474195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.563701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 733.523714] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 16:10:58 (1763154658) [ 735.244316] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 737.122792] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 16:11:01 (1763154661) [ 738.485866] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 740.028442] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 16:11:04 (1763154664) [ 741.450748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 743.059565] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 16:11:07 (1763154667) [ 744.699901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 746.426327] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 16:11:11 (1763154671) [ 748.150839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 749.876940] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 16:11:14 (1763154674) [ 750.246676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 750.296182] LNet: 50457:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 750.343512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 750.463479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 750.492993] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 751.217410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.29 [ 751.232792] LNet: Removed LNI 192.168.203.29@tcp [ 757.312652] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 16:11:21 (1763154681) [ 757.432163] LNet: 51151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.449119] LNetError: 51151:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.467282] LNet: Removed LNI 192.168.203.29@tcp [ 758.297110] Key type .llcrypt unregistered [ 758.298159] Key type ._llcrypt unregistered [ 763.665039] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 763.894670] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 763.897755] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 763.899980] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 766.792969] Key type ._llcrypt registered [ 766.794351] Key type .llcrypt registered [ 767.039922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.064864] alg: No test for adler32 (adler32-zlib) [ 768.371106] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.974862] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 16:11:39 (1763154699) [ 775.265611] LNet: 52061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.278500] LNetError: 52061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 775.311111] LNet: Removed LNI 10.1.2.3@tcp [ 775.985623] Key type .llcrypt unregistered [ 775.989366] Key type ._llcrypt unregistered [ 782.274497] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 782.559701] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 782.564555] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 782.569732] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 785.249847] Key type ._llcrypt registered [ 785.251931] Key type .llcrypt registered [ 785.391676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 785.406764] alg: No test for adler32 (adler32-zlib) [ 786.341284] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 792.868425] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 16:11:57 (1763154717) [ 793.020340] LNet: 52945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.024882] LNetError: 52945:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 793.048293] LNet: Removed LNI 10.1.2.3@tcp [ 793.584958] Key type .llcrypt unregistered [ 793.586942] Key type ._llcrypt unregistered [ 798.770104] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 798.953420] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 798.955706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 798.960444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 801.554848] Key type ._llcrypt registered [ 801.561929] Key type .llcrypt registered [ 801.770264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 801.785081] alg: No test for adler32 (adler32-zlib) [ 802.757560] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.517606] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 16:12:13 (1763154733) [ 808.631809] LNet: 53829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.640468] LNetError: 53829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.655485] LNet: Removed LNI 10.1.2.3@tcp [ 809.349445] Key type .llcrypt unregistered [ 809.351239] Key type ._llcrypt unregistered [ 814.528577] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.725553] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.729199] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.732607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.420294] Key type ._llcrypt registered [ 817.421607] Key type .llcrypt registered [ 817.687854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.704679] alg: No test for adler32 (adler32-zlib) [ 818.699118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.780900] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 826.631759] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 16:12:31 (1763154751) [ 827.466870] LNet: 55063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 827.470554] LNetError: 55063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 827.488451] LNet: Removed LNI 10.1.2.3@tcp [ 828.250386] Key type .llcrypt unregistered [ 828.252343] Key type ._llcrypt unregistered [ 833.090271] Key type ._llcrypt registered [ 833.092316] Key type .llcrypt registered [ 833.332573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.356491] alg: No test for adler32 (adler32-zlib) [ 834.327744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 834.351851] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 840.595612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 841.363581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_interrupt -r 1 [ 841.438240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_interrupt -r 1 [ 841.504337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 841.530274] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 841.869734] LNet: 55489:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 841.939862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 841.982547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 842.014244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 842.533494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_dropped -r 1 [ 842.566750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_dropped -r 1 [ 842.624989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 842.647032] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 842.937620] LNet: 55586:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 843.003524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 843.035843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 843.084203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 843.581488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_aborted -r 1 [ 843.616477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_aborted -r 1 [ 843.659350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 843.682142] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 843.959235] LNet: 55684:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 844.023593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 844.051637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 844.084628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 844.477435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_no_route -r 1 [ 844.503707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_no_route -r 1 [ 844.532505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 844.875190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 844.905948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 844.926518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 845.341689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_timeout -r 1 [ 845.370757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_timeout -r 1 [ 845.401380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 845.743855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 845.773970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 845.805251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 846.167970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_error -r 1 [ 846.196991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_error -r 1 [ 846.227270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 846.241862] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 846.245725] LNet: Skipped 2 previous similar messages [ 846.541416] LNet: 55975:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 846.546969] LNet: 55975:0:(api-ni.c:339:recovery_interval_set()) Skipped 2 previous similar messages [ 846.611748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 846.686568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 846.705913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 846.985719] LNet: 56011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.007972] LNetError: 56011:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.036314] LNet: Removed LNI 192.168.203.29@tcp [ 847.617557] Key type .llcrypt unregistered [ 847.621067] Key type ._llcrypt unregistered [ 862.116305] Key type ._llcrypt registered [ 862.117977] Key type .llcrypt registered [ 863.593681] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 16:13:07 (1763154787) [ 864.654285] Key type .llcrypt unregistered [ 864.656061] Key type ._llcrypt unregistered [ 871.320217] Key type ._llcrypt registered [ 871.327691] Key type .llcrypt registered [ 871.579550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 871.608607] alg: No test for adler32 (adler32-zlib) [ 872.732801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 872.783422] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 880.633709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 889.477574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.545540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 889.605854] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 891.160978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_interrupt -r 1 [ 891.246920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 1 [ 891.330109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_interrupt -r 1 [ 891.374777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 1 [ 891.421341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_interrupt -r 1 [ 891.469521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 1 [ 891.504570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_interrupt -r 1 [ 891.533801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 1 [ 891.569584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 891.587826] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 891.594989] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 891.917898] LNet: 57465:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 892.008174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 892.048274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 892.119557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 892.803841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_dropped -r 1 [ 892.849518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 1 [ 892.891865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_dropped -r 1 [ 892.932177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 1 [ 892.968924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_dropped -r 1 [ 893.007684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 1 [ 893.036452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_dropped -r 1 [ 893.072813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 1 [ 893.119853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 893.137832] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 893.142277] LNet: Skipped 2 previous similar messages [ 893.445196] LNet: 57574:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 893.531226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 893.563794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 893.596706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 894.043932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_aborted -r 1 [ 894.074894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 1 [ 894.099456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_aborted -r 1 [ 894.126354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 1 [ 894.164524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_aborted -r 1 [ 894.192961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 1 [ 894.216693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_aborted -r 1 [ 894.251213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 1 [ 894.275348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 894.291519] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 894.296096] LNet: Skipped 2 previous similar messages [ 894.523401] LNet: 57684:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 894.582183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 894.630144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 894.661813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 895.087398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_no_route -r 1 [ 895.110239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 1 [ 895.152682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_no_route -r 1 [ 895.185548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 1 [ 895.227769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_no_route -r 1 [ 895.258786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 1 [ 895.283386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_no_route -r 1 [ 895.307935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 1 [ 895.341341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 895.716369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 895.768862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 895.816840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 896.276873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_timeout -r 1 [ 896.304477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 1 [ 896.326306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_timeout -r 1 [ 896.358827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 1 [ 896.397819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_timeout -r 1 [ 896.435502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 1 [ 896.472461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_timeout -r 1 [ 896.530733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 1 [ 896.591281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 896.621321] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 896.628624] LNet: Skipped 5 previous similar messages [ 896.878751] LNet: 57902:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 896.883551] LNet: 57902:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 896.947099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 896.975781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 897.000873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 897.373471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_error -r 1 [ 897.396787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_error -r 1 [ 897.416856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_error -r 1 [ 897.450303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_error -r 1 [ 897.472394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_error -r 1 [ 897.497195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_error -r 1 [ 897.541138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_error -r 1 [ 897.571908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_error -r 1 [ 897.611145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 897.959264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 897.990471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 898.017913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 901.504485] LNet: 58082:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.507803] LNetError: 58082:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.512291] LNet: Removed LNI 192.168.203.29@tcp [ 901.526701] LNet: Removed LNI 192.168.203.29@tcp1 [ 901.759951] Key type .llcrypt unregistered [ 901.761432] Key type ._llcrypt unregistered [ 914.106336] Key type ._llcrypt registered [ 914.109242] Key type .llcrypt registered [ 915.245671] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 16:13:59 (1763154839) [ 916.260533] Key type .llcrypt unregistered [ 916.262468] Key type ._llcrypt unregistered [ 921.297746] Key type ._llcrypt registered [ 921.299495] Key type .llcrypt registered [ 921.558815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 921.588575] alg: No test for adler32 (adler32-zlib) [ 922.542714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 922.594840] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 929.151258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 929.784713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_dropped -r 1 [ 929.823974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_dropped -r 1 [ 929.866315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 929.886411] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 930.171423] LNet: 59318:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 930.238693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.276035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.321240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 930.858163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_error -r 1 [ 930.891939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_error -r 1 [ 930.933780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 930.963340] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 931.279422] LNet: 59415:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 931.339834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 931.366291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 931.388951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 931.908379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_timeout -r 1 [ 931.960241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_timeout -r 1 [ 931.994968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 932.005722] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 932.334722] LNet: 59513:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 932.398242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.424120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.452781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 932.683818] LNet: 59549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.692204] LNetError: 59549:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.715684] LNet: Removed LNI 192.168.203.29@tcp [ 933.059595] Key type .llcrypt unregistered [ 933.061334] Key type ._llcrypt unregistered [ 946.563223] Key type ._llcrypt registered [ 946.568164] Key type .llcrypt registered [ 947.579497] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 16:14:32 (1763154872) [ 948.411651] Key type .llcrypt unregistered [ 948.414172] Key type ._llcrypt unregistered [ 953.043086] Key type ._llcrypt registered [ 953.054144] Key type .llcrypt registered [ 953.251620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 953.266074] alg: No test for adler32 (adler32-zlib) [ 954.164640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 954.187849] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 960.990782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 968.237723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 968.281954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 968.305811] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 969.607860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_dropped -r 1 [ 969.626501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 1 [ 969.677767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_dropped -r 1 [ 969.703670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 1 [ 969.767994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_dropped -r 1 [ 969.798865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 1 [ 969.828865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_dropped -r 1 [ 969.861324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 1 [ 969.900679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 969.923360] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 969.931096] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 970.221625] LNet: 61001:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 970.277800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.298666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 970.324764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.365821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.915589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_error -r 1 [ 970.954765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_error -r 1 [ 970.994736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_error -r 1 [ 971.035387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_error -r 1 [ 971.069917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_error -r 1 [ 971.097228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_error -r 1 [ 971.134584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_error -r 1 [ 971.166193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_error -r 1 [ 971.195627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 971.217989] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 971.226499] LNet: Skipped 2 previous similar messages [ 971.608220] LNet: 61113:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 971.681284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.722849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.791536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 971.807865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 972.483513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_timeout -r 1 [ 972.537167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_timeout -r 1 [ 972.573973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_timeout -r 1 [ 972.614078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_timeout -r 1 [ 972.637925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_timeout -r 1 [ 972.683424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_timeout -r 1 [ 972.721867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_timeout -r 1 [ 972.763942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_timeout -r 1 [ 972.797300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.129@tcp [ 972.816243] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 973.180339] LNet: 61224:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 973.275323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 973.324204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 973.359422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 973.400491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 977.194771] LNet: 61297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 977.201993] LNetError: 61297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.208146] LNet: Removed LNI 192.168.203.29@tcp [ 978.210152] LNetError: 61297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 978.238325] LNet: Removed LNI 192.168.203.29@tcp1 [ 978.686597] Key type .llcrypt unregistered [ 978.689362] Key type ._llcrypt unregistered [ 993.134579] Key type ._llcrypt registered [ 993.136706] Key type .llcrypt registered [ 994.248066] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 16:15:18 (1763154918) [ 994.692217] Key type .llcrypt unregistered [ 994.695592] Key type ._llcrypt unregistered [ 999.325094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 999.442414] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 999.448248] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 999.454172] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1002.073725] Key type ._llcrypt registered [ 1002.079187] Key type .llcrypt registered [ 1002.300640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.341617] alg: No test for adler32 (adler32-zlib) [ 1003.256656] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1003.290179] LNet: 62385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.295965] LNetError: 62385:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.310596] LNet: Removed LNI 192.168.203.29@tcp [ 1004.138859] Key type .llcrypt unregistered [ 1004.153531] Key type ._llcrypt unregistered [ 1009.793812] Key type ._llcrypt registered [ 1009.796889] Key type .llcrypt registered [ 1010.047380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1010.078477] alg: No test for adler32 (adler32-zlib) [ 1011.041542] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1011.044257] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1011.082114] LNet: 62654:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.086526] LNetError: 62654:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.090686] LNet: Removed LNI 192.168.203.29@tcp [ 1011.128635] LNet: Removed LNI 10.1.2.3@tcp1 [ 1011.715756] Key type .llcrypt unregistered [ 1011.718988] Key type ._llcrypt unregistered [ 1016.136906] Key type ._llcrypt registered [ 1016.143300] Key type .llcrypt registered [ 1016.362837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1016.389923] alg: No test for adler32 (adler32-zlib) [ 1017.345722] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1017.348510] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1017.401070] LNet: 62924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.408371] LNetError: 62924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1017.418224] LNet: Removed LNI 192.168.203.29@tcp [ 1017.427908] LNet: Removed LNI 10.1.2.3@tcp [ 1018.171974] Key type .llcrypt unregistered [ 1018.174888] Key type ._llcrypt unregistered [ 1023.434064] Key type ._llcrypt registered [ 1023.435913] Key type .llcrypt registered [ 1023.658790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1023.667778] alg: No test for adler32 (adler32-zlib) [ 1024.572678] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1024.574624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1024.614756] LNet: 63194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.621834] LNetError: 63194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.625074] LNet: Removed LNI 192.168.203.29@tcp [ 1024.635986] LNet: Removed LNI 10.1.2.3@tcp [ 1025.209268] Key type .llcrypt unregistered [ 1025.216722] Key type ._llcrypt unregistered [ 1030.424139] Key type ._llcrypt registered [ 1030.429098] Key type .llcrypt registered [ 1030.584193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.600119] alg: No test for adler32 (adler32-zlib) [ 1031.533905] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1031.536549] LNetError: 63454:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1031.541779] LNetError: Error -100 starting up LNI tcp [ 1031.557813] LNet: Removed LNI 192.168.203.29@tcp [ 1032.134444] Key type .llcrypt unregistered [ 1032.136771] Key type ._llcrypt unregistered [ 1041.816120] Key type ._llcrypt registered [ 1041.818059] Key type .llcrypt registered [ 1042.950148] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 16:16:07 (1763154967) [ 1043.812355] Key type .llcrypt unregistered [ 1043.813770] Key type ._llcrypt unregistered [ 1048.927741] Key type ._llcrypt registered [ 1048.928854] Key type .llcrypt registered [ 1049.184789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1049.214454] alg: No test for adler32 (adler32-zlib) [ 1050.096870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1050.124932] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1056.393680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1057.103311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e network_timeout -r 1 [ 1057.125677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e network_timeout -r 1 [ 1057.158421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1057.179271] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1057.449634] LNet: 64711:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.502862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.531903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.729612] LNet: 64745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1057.734595] LNetError: 64745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1057.754802] LNet: Removed LNI 192.168.203.29@tcp [ 1058.100223] Key type .llcrypt unregistered [ 1058.105942] Key type ._llcrypt unregistered [ 1075.611065] Key type ._llcrypt registered [ 1075.612629] Key type .llcrypt registered [ 1075.870661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.905920] alg: No test for adler32 (adler32-zlib) [ 1076.870701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1076.892387] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1082.814705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1089.683741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.737775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1089.775316] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1091.050595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e network_timeout -r 1 [ 1091.091322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e network_timeout -r 1 [ 1091.144304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e network_timeout -r 1 [ 1091.177153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e network_timeout -r 1 [ 1091.214631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e network_timeout -r 1 [ 1091.281364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e network_timeout -r 1 [ 1091.331653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e network_timeout -r 1 [ 1091.375067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e network_timeout -r 1 [ 1091.428777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1091.452484] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1091.734339] LNet: 65609:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.817916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.861234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.263893] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1095.330113] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 1095.553433] LNet: 65677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.558964] LNetError: 65677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.608206] LNet: Removed LNI 192.168.203.29@tcp [ 1096.610711] LNetError: 65677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1096.626818] LNet: Removed LNI 192.168.203.29@tcp1 [ 1097.092847] Key type .llcrypt unregistered [ 1097.094259] Key type ._llcrypt unregistered [ 1109.791123] Key type ._llcrypt registered [ 1109.792378] Key type .llcrypt registered [ 1110.789819] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 16:17:15 (1763155035) [ 1111.129921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1111.144756] alg: No test for adler32 (adler32-zlib) [ 1112.072405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1112.116387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1112.158586] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1112.778710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1112.806429] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1113.667742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1113.727256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1113.761303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1113.790403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1113.815561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1113.843093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1113.870716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.29@tcp [ 1113.888326] LNetError: 66776:0:(api-ni.c:4321:lnet_ni_set_healthv()) manually adding local NI 192.168.203.29@tcp to recovery [ 1114.145738] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1114.151216] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 1116.197161] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1116.323384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1120.288392] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1120.828554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1120.878869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1120.941047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1120.964353] LNet: 66883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1120.974508] LNetError: 66883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1120.980657] LNet: Removed LNI 192.168.203.29@tcp [ 1120.998224] LNet: Removed LNI 192.168.203.29@tcp1 [ 1121.017883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.074398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1121.099977] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1121.705944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1121.724864] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1122.445357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1122.476629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.530791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1122.565374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1122.596487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1122.634359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1122.683898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1122.728471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.29@tcp [ 1122.767676] LNetError: 67155:0:(api-ni.c:4321:lnet_ni_set_healthv()) manually adding local NI 192.168.203.29@tcp to recovery [ 1123.104325] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1126.234186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1129.249314] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1129.257960] LNet: Skipped 1 previous similar message [ 1133.933490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1133.967955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.017723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1139.595628] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 16:17:44 (1763155064) [ 1139.799846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1139.829771] LNet: 67872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.848568] LNetError: 67872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.852171] LNetError: 67872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1139.860995] LNet: Removed LNI 192.168.203.29@tcp [ 1139.877760] LNet: Removed LNI 192.168.203.29@tcp1 [ 1139.902732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.971539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1140.001030] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1140.712493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1140.737613] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1141.444837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1141.502785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1141.531196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1141.567988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1141.613970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1141.650838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1141.699813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.29@tcp --health 0 [ 1141.819761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1141.984134] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.29@tcp [ 1143.008623] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1143.019981] LNet: Skipped 1 previous similar message [ 1157.864946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1157.999826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1158.031854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1158.075110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.29@tcp --health 500 [ 1158.163662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1164.718407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1164.767618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1164.788460] LNet: 68372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.795128] LNetError: 68372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1164.802552] LNetError: 68372:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1164.805715] LNet: Removed LNI 192.168.203.29@tcp [ 1164.826984] LNet: Removed LNI 192.168.203.29@tcp1 [ 1164.850971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.907673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1164.931647] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1165.697425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.725912] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1166.698726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1166.730961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.819901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1166.875296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1166.917422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1166.991854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1167.074948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1167.102646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.29@tcp --health 0 [ 1168.994395] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1169.001232] LNet: Skipped 3 previous similar messages [ 1169.509532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1178.677990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1178.715865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.766180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.100331] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 16:18:29 (1763155109) [ 1184.525154] LNet: 69421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1184.537386] LNetError: 69421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1184.539847] LNetError: 69421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1184.553359] LNet: Removed LNI 192.168.203.29@tcp [ 1184.578211] LNet: Removed LNI 192.168.203.29@tcp1 [ 1185.157718] Key type .llcrypt unregistered [ 1185.161345] Key type ._llcrypt unregistered [ 1189.241336] Key type ._llcrypt registered [ 1189.244126] Key type .llcrypt registered [ 1189.449826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1189.460385] alg: No test for adler32 (adler32-zlib) [ 1190.356464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1190.383099] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1197.040554] Lustre: DEBUG MARKER: Initial discovery [ 1197.081911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.129@tcp [ 1199.132793] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1199.173841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1199.226167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.129@tcp [ 1199.244995] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1199.266530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1199.300342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.129@tcp -r 1 -m GET -l 3 [ 1199.361038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.129@tcp -r 1 -m GET -e local_error [ 1199.424861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.129@tcp -r 1 -m PUT -l 6 [ 1200.432977] Lustre: DEBUG MARKER: Force oleg329-client.virtnet to discover 192.168.203.129@tcp (in background) [ 1200.576816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.129@tcp [ 1201.589631] Lustre: DEBUG MARKER: Force oleg329-server to discover 192.168.203.29@tcp [ 1203.599769] Lustre: DEBUG MARKER: Wait for 70179 [ 1203.617475] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1205.056085] Lustre: DEBUG MARKER: Finished wait on 70179 [ 1205.207759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1205.237151] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1205.242889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.526689] LNet: 70626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.532804] LNetError: 70626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.551724] LNet: Removed LNI 192.168.203.29@tcp [ 1205.986765] Key type .llcrypt unregistered [ 1205.988299] Key type ._llcrypt unregistered [ 1218.203471] Key type ._llcrypt registered [ 1218.209394] Key type .llcrypt registered [ 1219.389754] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 16:19:04 (1763155144) [ 1220.079563] Key type .llcrypt unregistered [ 1220.081753] Key type ._llcrypt unregistered [ 1225.016923] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1225.094158] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1225.096877] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1225.099407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1227.615956] Key type ._llcrypt registered [ 1227.617800] Key type .llcrypt registered [ 1227.788588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.801887] alg: No test for adler32 (adler32-zlib) [ 1228.740826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1228.822236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1228.862833] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1229.560072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1229.577124] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1237.570734] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 16:19:22 (1763155162) [ 1237.775781] LNet: 72690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.786665] LNetError: 72690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.796738] LNet: Removed LNI 192.168.203.29@tcp [ 1237.807253] LNet: Removed LNI 10.1.2.3@tcp [ 1238.450263] Key type .llcrypt unregistered [ 1238.455405] Key type ._llcrypt unregistered [ 1243.363489] Key type ._llcrypt registered [ 1243.365016] Key type .llcrypt registered [ 1243.581824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1243.604361] alg: No test for adler32 (adler32-zlib) [ 1244.500509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1244.556877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1244.576776] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1245.168774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1245.188617] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1245.969767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.29@tcp [ 1247.072141] LNetError: Unexpected error -22 connecting to 192.168.203.29@tcp at host 192.168.203.29:988 [ 1251.133318] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 16:19:36 (1763155176) [ 1251.271102] LNet: 73816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.274929] LNetError: 73816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.320146] LNet: Removed LNI 192.168.203.29@tcp [ 1252.321811] LNetError: 73816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1252.348503] LNet: Removed LNI 10.1.2.3@tcp [ 1252.907485] Key type .llcrypt unregistered [ 1252.909731] Key type ._llcrypt unregistered [ 1257.188697] Key type ._llcrypt registered [ 1257.191345] Key type .llcrypt registered [ 1257.369802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1257.380500] alg: No test for adler32 (adler32-zlib) [ 1258.292668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1258.369169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1258.393251] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1258.994820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1259.016413] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1259.798843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.29@tcp1 --nid 192.168.203.29@tcp2 [ 1263.602396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp1 192.168.203.29@tcp2 [ 1263.617964] LNetError: 74647:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.29@tcp1 to 192.168.203.29@tcp2 - Route table may be misconfigured [ 1263.634746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp2 192.168.203.29@tcp1 [ 1268.918257] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 16:19:53 (1763155193) [ 1269.176592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.203036] LNet: 75220:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.206276] LNetError: 75220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.240692] LNet: Removed LNI 192.168.203.29@tcp1 [ 1270.242790] LNetError: 75220:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1270.264787] LNet: Removed LNI 192.168.203.29@tcp2 [ 1270.290906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.334118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1270.359527] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1270.972935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1270.990081] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1271.766206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1271.796739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.29@tcp [ 1271.833945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp1 192.168.203.29@tcp1 [ 1271.869782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.29@tcp -d 192.168.203.29@tcp -e network_timeout [ 1271.897941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -e network_timeout [ 1271.942416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -e network_timeout [ 1271.971763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -e network_timeout [ 1272.003862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1272.022823] LNet: There was an unexpected network error while writing to 192.168.203.29: rc = -22 [ 1272.035862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1272.113763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1272.188915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1272.354937] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 1277.510725] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 16:20:02 (1763155202) [ 1277.609988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.629958] LNet: 76076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1277.633670] LNetError: 76076:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1277.638415] LNet: Removed LNI 192.168.203.29@tcp [ 1278.696895] LNet: Removed LNI 192.168.203.29@tcp1 [ 1278.717619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.801664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1279.103575] LNet: 76118:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.492461] Key type .llcrypt unregistered [ 1279.504111] Key type ._llcrypt unregistered [ 1290.284355] Key type ._llcrypt registered [ 1290.285682] Key type .llcrypt registered [ 1291.382510] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 16:20:16 (1763155216) [ 1291.822174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1291.833210] alg: No test for adler32 (adler32-zlib) [ 1292.773069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.353182] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1296.076926] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 16:20:20 (1763155220) [ 1296.168026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1296.194543] LNet: 77207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1296.214075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.331401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1296.373070] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1297.188656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1297.202727] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1298.212536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1298.245727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp1 [ 1298.282217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp1 [ 1304.097701] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 16:20:28 (1763155228) [ 1305.817915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg329-server [ 1307.207658] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 16:20:32 (1763155232) [ 1308.683864] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg329-server [ 1310.198105] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 16:20:35 (1763155235) [ 1311.788952] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg329-server [ 1313.387266] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 16:20:38 (1763155238) [ 1314.816481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg329-server [ 1316.239899] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 16:20:41 (1763155241) [ 1318.047652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg329-server [ 1319.506756] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 16:20:44 (1763155244) [ 1321.199857] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg329-server [ 1323.095294] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 16:20:47 (1763155247) [ 1325.038878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg329-server [ 1326.652059] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 16:20:51 (1763155251) [ 1328.006764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg329-server [ 1329.632581] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 16:20:54 (1763155254) [ 1331.552645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg329-server [ 1333.108217] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 16:20:57 (1763155257) [ 1333.388104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1333.413673] LNet: 81089:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.420344] LNetError: 81089:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.424918] LNet: Removed LNI 192.168.203.29@tcp [ 1334.444184] LNet: Removed LNI 192.168.203.29@tcp1 [ 1334.482160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1334.583442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1334.607083] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1335.356025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1335.393814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1335.557715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1335.587777] LNet: 81237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1335.600305] LNetError: 81237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1335.606808] LNetError: 81237:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1335.623632] LNet: Removed LNI 192.168.203.29@tcp [ 1335.639231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1335.694244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1335.740456] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1336.633280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1336.852911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1336.902898] LNetError: 81382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1336.929677] LNet: Removed LNI 192.168.203.29@tcp [ 1336.980370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1337.108655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1337.146065] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1337.878951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1337.975719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1338.221605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1338.248989] LNet: 81529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1338.257922] LNet: 81529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1339.309782] LNet: Removed LNI 192.168.203.29@tcp [ 1339.340443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1339.387817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1339.417355] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1340.269332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1340.490943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1340.524145] LNetError: 81675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1340.537479] LNetError: 81675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1340.565103] LNet: Removed LNI 192.168.203.29@tcp [ 1340.593545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1340.655214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1340.703157] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1341.536112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1341.624775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1341.858958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1342.951863] LNet: Removed LNI 192.168.203.29@tcp [ 1342.980944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1343.030399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1343.065216] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1343.879341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1344.127179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1344.147520] LNet: 81967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1344.159819] LNet: 81967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1345.197053] LNet: Removed LNI 192.168.203.29@tcp [ 1345.220494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.257740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.273410] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1345.926721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1346.022085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1346.272913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1346.298714] LNetError: 82115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.304012] LNetError: 82115:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1347.367779] LNet: Removed LNI 192.168.203.29@tcp [ 1347.415399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1347.489986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1347.521310] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1348.279229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1348.544852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.607691] LNet: Removed LNI 192.168.203.29@tcp [ 1349.647655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.681647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1349.716596] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1350.352295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1350.435378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1350.701064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1351.789849] LNet: Removed LNI 192.168.203.29@tcp [ 1351.832974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.916847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1351.950972] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1352.737506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1353.024822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.051239] LNet: 82553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1353.062599] LNet: 82553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1354.086794] LNet: Removed LNI 192.168.203.29@tcp [ 1354.119204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.180691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.209746] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1354.891055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1354.995834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1355.299858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.327656] LNetError: 82700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.330542] LNetError: 82700:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1356.397310] LNet: Removed LNI 192.168.203.29@tcp [ 1356.425637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.501345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1356.555546] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1357.379448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1357.731957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1358.823721] LNet: Removed LNI 192.168.203.29@tcp [ 1358.862689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.938484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1358.958625] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1359.634667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1359.698638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.731802] LNet: Removed LNI 192.168.203.29@tcp [ 1359.750541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1359.810243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1359.834339] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1360.640296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1366.923177] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 16:21:31 (1763155291) [ 1367.108655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1367.150044] LNet: Removed LNI 192.168.203.29@tcp [ 1367.183275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1367.229167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1367.251472] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 1368.374066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.416336] LNet: Removed LNI 192.168.203.29@tcp231 [ 1368.443034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.531060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1368.569509] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 1368.574528] LNet: 83829:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1369.060306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1369.092685] LNet: Removed LNI 192.168.203.29@tcp231 [ 1369.114346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1369.141206] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 1370.334922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.372467] LNet: 83973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.381286] LNet: 83973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1370.398634] LNet: Removed LNI 192.168.203.29@tcp231 [ 1370.416794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.499980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1370.558631] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/0] [ 1370.566183] LNet: 83981:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1370.974415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1370.998614] LNet: Removed LNI 192.168.203.29@tcp231 [ 1371.018498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1371.057653] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/0] [ 1372.016179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1372.028847] LNetError: 84124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1372.033055] LNetError: 84124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1372.045903] LNet: Removed LNI 192.168.203.29@tcp231 [ 1372.060146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1372.155802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1372.193450] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/60] [ 1372.197476] LNet: 84132:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1372.559802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1372.593874] LNet: Removed LNI 192.168.203.29@tcp231 [ 1372.604907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1372.625920] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/60] [ 1373.600745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.648345] LNet: Removed LNI 192.168.203.29@tcp231 [ 1373.668206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.748476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1373.777268] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 1374.201036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1374.226552] LNet: Removed LNI 192.168.203.29@tcp231 [ 1374.245697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1374.301784] LNet: Added LNI 192.168.203.29@tcp231 [8/256/0/180] [ 1381.288563] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 16:21:46 (1763155306) [ 1381.389081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.444623] LNet: Removed LNI 192.168.203.29@tcp231 [ 1381.476255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.546779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1381.574485] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 1382.364949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1382.412123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1382.440594] LNet: Removed LNI 192.168.203.29@tcp232 [ 1382.458895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1382.531440] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 1383.325733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1383.371161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1383.415499] LNet: Removed LNI 192.168.203.29@tcp232 [ 1383.439625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1383.475422] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 1384.234701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1384.280360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1384.313962] LNet: Removed LNI 192.168.203.29@tcp232 [ 1384.334938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1384.368336] LNet: Added LNI 192.168.203.29@tcp232 [8/256/0/180] [ 1385.064509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1385.092456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1385.122884] LNet: Removed LNI 192.168.203.29@tcp232 [ 1385.134302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1390.699318] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 16:21:55 (1763155315) [ 1391.613209] Key type .llcrypt unregistered [ 1391.615381] Key type ._llcrypt unregistered [ 1396.531848] Key type ._llcrypt registered [ 1396.533395] Key type .llcrypt registered [ 1396.793143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1396.832652] alg: No test for adler32 (adler32-zlib) [ 1397.748895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1397.777675] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1404.984686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1412.129477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.164474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1412.187066] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1413.611527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_interrupt -r 2 [ 1413.659746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 2 [ 1413.691669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_interrupt -r 2 [ 1413.725648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 2 [ 1413.763346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_interrupt -r 2 [ 1413.802639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_interrupt -r 2 [ 1413.845216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_interrupt -r 2 [ 1413.885093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_interrupt -r 2 [ 1413.921826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 1413.942664] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1413.950824] LNet: 1 local NIs in recovery (showing 1): 192.168.203.29@tcp [ 1414.384360] LNet: 86773:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.454877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1414.488428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.010375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_dropped -r 2 [ 1415.037862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 2 [ 1415.071528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_dropped -r 2 [ 1415.139935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 2 [ 1415.188743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_dropped -r 2 [ 1415.236659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_dropped -r 2 [ 1415.316810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_dropped -r 2 [ 1415.359757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_dropped -r 2 [ 1415.407279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 1415.436684] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1415.810635] LNet: 86881:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1415.892191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1415.960595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1416.401576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_aborted -r 2 [ 1416.434984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 2 [ 1416.460315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_aborted -r 2 [ 1416.499634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 2 [ 1416.535943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_aborted -r 2 [ 1416.588359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_aborted -r 2 [ 1416.641860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_aborted -r 2 [ 1416.701416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_aborted -r 2 [ 1416.773427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 1416.805491] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1417.199337] LNet: 86990:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1417.271454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1417.320401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1417.836575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_no_route -r 2 [ 1417.876334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 2 [ 1417.923657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_no_route -r 2 [ 1417.957801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 2 [ 1418.004754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_no_route -r 2 [ 1418.032812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_no_route -r 2 [ 1418.063259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_no_route -r 2 [ 1418.092697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_no_route -r 2 [ 1418.131290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 1418.568963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1418.596887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1419.022047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e local_timeout -r 2 [ 1419.042608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 2 [ 1419.068562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e local_timeout -r 2 [ 1419.093878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 2 [ 1419.123855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e local_timeout -r 2 [ 1419.148652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e local_timeout -r 2 [ 1419.218565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e local_timeout -r 2 [ 1419.264556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e local_timeout -r 2 [ 1419.320738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 1419.341808] LNet: There was an unexpected network error while writing to 192.168.203.129: rc = -22 [ 1419.345378] LNet: Skipped 1 previous similar message [ 1419.746443] LNet: 87206:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1419.760521] LNet: 87206:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1419.851866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1419.881420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1420.359587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp -m GET -e remote_dropped -r 2 [ 1420.391712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 2 [ 1420.421578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp -m GET -e remote_dropped -r 2 [ 1420.459015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 2 [ 1420.514612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp -m GET -e remote_dropped -r 2 [ 1420.553255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.129@tcp1 -m GET -e remote_dropped -r 2 [ 1420.589373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp -m GET -e remote_dropped -r 2 [ 1420.610667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.29@tcp1 -d 192.168.203.29@tcp1 -m GET -e remote_dropped -r 2 [ 1420.646271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.29@tcp 192.168.203.129@tcp [ 1420.678326] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 1421.102467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.128379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1425.449837] LNet: 87383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.459424] LNetError: 87383:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.471133] LNet: Removed LNI 192.168.203.29@tcp [ 1425.484550] LNet: Removed LNI 192.168.203.29@tcp1 [ 1425.893852] Key type .llcrypt unregistered [ 1425.895175] Key type ._llcrypt unregistered [ 1439.452306] Key type ._llcrypt registered [ 1439.453894] Key type .llcrypt registered [ 1440.618748] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 16:22:45 (1763155365) [ 1440.996187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1441.017631] alg: No test for adler32 (adler32-zlib) [ 1441.949801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.013359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1442.051645] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1442.668883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1442.756868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1442.817570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1442.881702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1442.916261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1442.953621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1448.325723] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 16:22:53 (1763155373) [ 1448.519670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.544797] LNet: 88942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.548978] LNetError: 88942:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.563225] LNet: Removed LNI 192.168.203.29@tcp [ 1448.598423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.674263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.724623] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1454.818415] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 16:22:59 (1763155379) [ 1454.949076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.985319] LNet: 89646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.992946] LNetError: 89646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.001923] LNet: Removed LNI 192.168.203.29@tcp [ 1455.027210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.077529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.106132] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1460.948687] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 16:23:05 (1763155385) [ 1462.300806] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1464.138574] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 16:23:08 (1763155388) [ 1464.789751] LNet: 90681:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1464.793874] LNetError: 90681:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.806455] LNet: Removed LNI 192.168.203.29@tcp [ 1465.425591] Key type .llcrypt unregistered [ 1465.427138] Key type ._llcrypt unregistered [ 1470.362772] Key type ._llcrypt registered [ 1470.364268] Key type .llcrypt registered [ 1470.637330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1470.653230] alg: No test for adler32 (adler32-zlib) [ 1471.590554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1471.648353] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1477.525644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1481.662970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.129@tcp [ 1481.952896] LNet: 91093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1481.959069] LNetError: 91093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1481.969865] LNet: Removed LNI 192.168.203.29@tcp [ 1482.266301] Key type .llcrypt unregistered [ 1482.268692] Key type ._llcrypt unregistered [ 1491.065119] Key type ._llcrypt registered [ 1491.066819] Key type .llcrypt registered [ 1492.149184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1493.564886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1495.021664] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 16:23:39 (1763155419) [ 1495.491429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1495.508369] alg: No test for adler32 (adler32-zlib) [ 1496.447145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1496.550416] LNet: 92178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1497.192659] Key type .llcrypt unregistered [ 1497.195243] Key type ._llcrypt unregistered [ 1503.032242] Key type ._llcrypt registered [ 1503.033836] Key type .llcrypt registered [ 1503.334466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1503.362951] alg: No test for adler32 (adler32-zlib) [ 1504.331737] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1504.334203] LNet: 92431:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1504.407875] LNetError: 92443:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.32@tcp is being used as a gateway but routing feature is not turned on [ 1504.429494] LNetError: 92443:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.32@tcp has gone from up to down [ 1504.930778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.960725] LNet: 92489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.977204] LNetError: 92489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.005152] LNet: Removed LNI 192.168.203.29@tcp [ 1505.039644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.089925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1505.129608] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1505.138777] LNet: 92496:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1505.143922] LNet: 92496:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1505.148121] LNetError: 92496:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.31@tcp is a gateway. Can not delete it [ 1505.170330] LNetError: 92496:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.203.31@tcp is a router. Can not be deleted [ 1505.190514] LNet: 92496:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1505.194235] LNetError: 92493:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.34@tcp is being used as a gateway but routing feature is not turned on [ 1505.237237] LNetError: 92493:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 1505.248774] LNetError: 92493:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.34@tcp has gone from up to down [ 1505.258838] LNetError: 92493:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1511.805205] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 16:23:56 (1763155436) [ 1513.570415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg329-server [ 1515.100930] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 16:23:59 (1763155439) [ 1516.838448] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg329-server [ 1518.595301] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 16:24:03 (1763155443) [ 1520.671598] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1522.624182] Key type lgssc registered [ 1523.648512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1542.795836] Key type lgssc unregistered [ 1543.123556] LNet: 94915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1543.131894] LNetError: 94915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1544.189827] LNet: Removed LNI 192.168.203.29@tcp [ 1544.834820] Key type .llcrypt unregistered [ 1544.838996] Key type ._llcrypt unregistered [ 1557.946474] Key type ._llcrypt registered [ 1557.948226] Key type .llcrypt registered [ 1559.152564] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 16:24:43 (1763155483) [ 1559.420887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1559.437286] alg: No test for adler32 (adler32-zlib) [ 1560.342888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.412664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.465127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.540305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1560.597425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1567.135474] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 16:24:51 (1763155491) [ 1567.320813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.363946] LNet: 96317:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1567.422401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.481808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.517650] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1568.399842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.428805] LNet: 96463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.437163] LNetError: 96463:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.448821] LNet: Removed LNI 192.168.203.29@tcp [ 1568.468895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.519225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1568.547393] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1568.967631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1568.991552] LNetError: 96491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.010604] LNet: Removed LNI 192.168.203.29@tcp [ 1569.032432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.070463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1569.093518] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1574.960376] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 16:24:59 (1763155499) [ 1575.227318] LNet: 97081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.237417] LNet: 97081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1575.241988] LNetError: 97081:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.261943] LNet: Removed LNI 192.168.203.29@tcp [ 1575.839720] Key type .llcrypt unregistered [ 1575.841167] Key type ._llcrypt unregistered [ 1580.712078] Key type ._llcrypt registered [ 1580.714721] Key type .llcrypt registered [ 1580.949679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1580.974749] alg: No test for adler32 (adler32-zlib) [ 1581.858854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.953088] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1581.955194] LNet: *** cfs_fail_loc=e003, val=100*** [ 1582.617747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1582.636653] LNet: *** cfs_fail_loc=e003, val=100*** [ 1582.638960] LNet: Skipped 2 previous similar messages [ 1582.647319] Ping reply received with mapping: [ 1582.652698] found metadata reply NI[192.168.203.29@tcp] [ 1582.654298] 1d cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1582.658264] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1582.660346] 5a 5a 5a 5a 5a 5a 5a 5a [ 1582.780653] LNet: 97475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.784721] LNetError: 97475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.845872] LNet: Removed LNI 192.168.203.29@tcp [ 1584.539393] Key type .llcrypt unregistered [ 1584.540652] Key type ._llcrypt unregistered [ 1593.446612] Key type ._llcrypt registered [ 1593.449169] Key type .llcrypt registered [ 1594.550974] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 16:25:19 (1763155519) [ 1595.203504] Key type .llcrypt unregistered [ 1595.205192] Key type ._llcrypt unregistered [ 1599.847388] Key type ._llcrypt registered [ 1599.849027] Key type .llcrypt registered [ 1600.006301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1600.023352] alg: No test for adler32 (adler32-zlib) [ 1601.089763] LNetError: 98511:0:(api-ni.c:2715:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1601.358591] Key type .llcrypt unregistered [ 1601.360107] Key type ._llcrypt unregistered [ 1605.556735] Key type ._llcrypt registered [ 1605.558181] Key type .llcrypt registered [ 1606.546209] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 16:25:31 (1763155531) [ 1607.107287] Key type .llcrypt unregistered [ 1607.112649] Key type ._llcrypt unregistered [ 1611.711134] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1611.713589] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1614.139512] Key type ._llcrypt registered [ 1614.141059] Key type .llcrypt registered [ 1614.317529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1614.336024] alg: No test for adler32 (adler32-zlib) [ 1615.259146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.306474] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1623.879663] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 16:25:48 (1763155548) [ 1624.071859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1624.100362] LNet: 100285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.108780] LNetError: 100285:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1624.128717] LNet: Removed LNI 10.1.2.3@tcp [ 1624.151273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.908838] Key type .llcrypt unregistered [ 1624.910567] Key type ._llcrypt unregistered [ 1629.818570] Key type ._llcrypt registered [ 1629.822652] Key type .llcrypt registered [ 1630.067572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1630.079958] alg: No test for adler32 (adler32-zlib) [ 1631.025994] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1631.076137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1631.214980] LNet: 100569:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.226708] LNetError: 100569:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.246875] LNet: Removed LNI 192.168.203.29@tcp [ 1631.717869] Key type .llcrypt unregistered [ 1631.723769] Key type ._llcrypt unregistered [ 1640.658291] Key type ._llcrypt registered [ 1640.664468] Key type .llcrypt registered [ 1641.746928] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 16:26:06 (1763155566) [ 1642.172520] Key type .llcrypt unregistered [ 1642.174264] Key type ._llcrypt unregistered [ 1647.094982] Key type ._llcrypt registered [ 1647.096577] Key type .llcrypt registered [ 1647.295994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1647.319467] alg: No test for adler32 (adler32-zlib) [ 1653.813459] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 16:26:18 (1763155578) [ 1653.914812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1653.977805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.092097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.119540] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1654.813296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1654.850226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1654.873964] LNet: Removed LNI 192.168.203.29@tcp [ 1655.058350] LNet: 102304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.472438] Key type .llcrypt unregistered [ 1655.475534] Key type ._llcrypt unregistered [ 1664.647171] Key type ._llcrypt registered [ 1664.648765] Key type .llcrypt registered [ 1665.732346] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 16:26:30 (1763155590) [ 1666.143213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1666.153859] alg: No test for adler32 (adler32-zlib) [ 1667.078305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.121572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1667.154868] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1667.797226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1667.825639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.29@tcp [ 1667.838738] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1667.841066] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1667.843239] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1667, 0, 0 [ 1667.851934] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1667.854772] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1667.862672] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1667.866855] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1667, 0, 0 [ 1667.872861] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1667.879525] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1667.883518] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1667.886559] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1667, 0, 1717 [ 1667.889805] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1667.891667] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1667.895717] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1667.900179] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1667, 0, 1717 [ 1667.905580] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1673.347927] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 16:26:38 (1763155598) [ 1673.767906] LNet: 103840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.778256] LNetError: 103840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.791837] LNet: Removed LNI 192.168.203.29@tcp [ 1675.432528] Key type .llcrypt unregistered [ 1675.434039] Key type ._llcrypt unregistered [ 1679.841857] Key type ._llcrypt registered [ 1679.843337] Key type .llcrypt registered [ 1679.997647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1680.016592] alg: No test for adler32 (adler32-zlib) [ 1680.894423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1680.921742] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1686.274730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1692.365813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1692.401079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1692.456896] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1693.395346] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1693.546653] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1693.554776] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1693.557313] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1695.882605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1695.940833] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1706.494398] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 16:27:11 (1763155631) [ 1706.628482] LNet: 105350:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.634552] LNetError: 105350:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.647518] LNet: Removed LNI 192.168.203.29@tcp [ 1706.649825] LNet: Removed LNI 192.168.203.29@tcp1 [ 1706.667784] LNet: Removed LNI 10.1.2.3@tcp99 [ 1707.327876] Key type .llcrypt unregistered [ 1707.334242] Key type ._llcrypt unregistered [ 1712.516850] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1712.521697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1714.992945] Key type ._llcrypt registered [ 1714.994202] Key type .llcrypt registered [ 1715.169348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1715.185390] alg: No test for adler32 (adler32-zlib) [ 1716.089692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.132832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.156089] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1716.848590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1716.863938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1718.813498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1718.847723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1718.880250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1718.908131] LNetError: 106094:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1718.951963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1718.983457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1719.247874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1719.318576] LNet: 106114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.324602] LNetError: 106114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1720.354253] LNet: Removed LNI 192.168.203.29@tcp [ 1720.356249] LNetError: 106114:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1720.378907] LNet: Removed LNI 10.1.2.3@tcp [ 1721.153651] Key type .llcrypt unregistered [ 1721.156832] Key type ._llcrypt unregistered [ 1730.609138] Key type ._llcrypt registered [ 1730.610965] Key type .llcrypt registered [ 1731.749542] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 16:27:36 (1763155656) [ 1732.068333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1732.083763] alg: No test for adler32 (adler32-zlib) [ 1733.043158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.102355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.126573] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1733.797414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.29@tcp [ 1733.895716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg329-client@tcp [ 1739.318501] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 16:27:44 (1763155664) [ 1739.586360] LNet: 107644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1739.597732] LNetError: 107644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.617145] LNet: Removed LNI 192.168.203.29@tcp [ 1740.185263] Key type .llcrypt unregistered [ 1740.186747] Key type ._llcrypt unregistered [ 1744.562503] Key type ._llcrypt registered [ 1744.570242] Key type .llcrypt registered [ 1744.754541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1744.778207] alg: No test for adler32 (adler32-zlib) [ 1745.705419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1745.730989] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1745.759936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1751.077432] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 16:27:55 (1763155675) [ 1752.392841] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1753.667461] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 16:27:58 (1763155678) [ 1755.183680] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1756.994670] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 16:28:01 (1763155681) [ 1757.235398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.257350] LNet: 109059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.264409] LNetError: 109059:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.284593] LNet: Removed LNI 192.168.203.29@tcp [ 1757.305406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.339781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1757.372293] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1758.094397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1758.135795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1763.945421] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 16:28:08 (1763155688) [ 1764.026685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.054710] LNet: 109763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.086516] LNetError: 109763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1765.100354] LNet: Removed LNI 192.168.203.29@tcp [ 1765.128698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.164546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1765.204256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1765.436750] LNet: 109804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.732524] Key type .llcrypt unregistered [ 1765.734538] Key type ._llcrypt unregistered [ 1776.555107] Key type ._llcrypt registered [ 1776.556968] Key type .llcrypt registered [ 1777.805836] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 16:28:22 (1763155702) [ 1778.345944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1778.356860] alg: No test for adler32 (adler32-zlib) [ 1779.346476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.454070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1779.516836] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1780.493532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1780.571361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1787.858569] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 16:28:32 (1763155712) [ 1787.947166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.974596] LNet: 111314:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.986608] LNetError: 111314:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.022383] LNet: Removed LNI 192.168.203.29@tcp [ 1788.061549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.161841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1788.203130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1793.807773] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 16:28:38 (1763155718) [ 1793.997316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1794.050605] LNet: 111902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1794.081565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1794.148919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1794.496061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1794.537262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1794.570058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1794.887405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1794.927458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1794.982205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.027965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.072643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1795.607884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1795.647148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1795.693972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1795.961959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1796.001210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1801.662827] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 16:28:46 (1763155726) [ 1801.781326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.802595] LNet: 112534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.805920] LNet: 112534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1801.896524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.966960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1802.001645] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1802.649490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1802.663056] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1803.382834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1803.434072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1803.659395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.682255] LNetError: 112815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.692939] LNet: Removed LNI 192.168.203.29@tcp [ 1803.706190] LNet: Removed LNI 192.168.203.29@tcp2 [ 1803.729415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.780755] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1803.789624] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1803.795705] LNet: 112821:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1804.105743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1804.269990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1804.304520] LNet: 112858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1804.308662] LNet: 112858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1805.348524] LNet: Removed LNI 192.168.203.29@tcp [ 1805.354690] LNetError: 112858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1805.370230] LNetError: 112858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1805.384203] LNet: Removed LNI 192.168.203.29@tcp2 [ 1805.426415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1805.493934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1805.521078] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1806.150804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1806.165322] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1806.807931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1806.829781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1806.859634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1807.015646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.032152] LNet: Removed LNI 192.168.203.29@tcp [ 1808.038348] LNetError: 113142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.040569] LNetError: 113142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1808.059636] LNet: Removed LNI 192.168.203.29@tcp2 [ 1808.093915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.199292] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1808.206969] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1808.213512] LNet: 113149:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1808.724349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1808.899422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.921089] LNet: 113186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.925849] LNet: 113186:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1808.934513] LNet: Removed LNI 192.168.203.29@tcp [ 1809.964646] LNet: Removed LNI 192.168.203.29@tcp2 [ 1809.982966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.067630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.088126] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1810.609674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1810.637415] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1811.332789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1811.384792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1811.570879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1812.641149] LNet: Removed LNI 192.168.203.29@tcp [ 1812.645764] LNetError: 113465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.649343] LNetError: 113465:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1812.661510] LNet: Removed LNI 192.168.203.29@tcp2 [ 1812.675956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.764173] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1812.773720] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1812.777773] LNet: 113471:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1813.124901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1813.278831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1814.304145] LNet: Removed LNI 192.168.203.29@tcp [ 1814.319212] LNet: Removed LNI 192.168.203.29@tcp2 [ 1814.368748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1814.419116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1814.435859] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1815.026902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1815.054978] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1815.866568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1815.926964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1815.988875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1816.193181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1817.248163] LNet: Removed LNI 192.168.203.29@tcp [ 1818.287991] LNet: Removed LNI 192.168.203.29@tcp2 [ 1818.318525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.383693] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1818.387549] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1818.396373] LNet: 113799:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1818.774532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1818.966966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1818.980933] LNet: 113836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.984420] LNet: 113836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1818.990467] LNet: Removed LNI 192.168.203.29@tcp [ 1820.008104] LNet: Removed LNI 192.168.203.29@tcp2 [ 1820.025649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1820.082882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1820.102932] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1820.609682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1820.626177] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1821.394614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1821.438928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1821.530626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1821.979470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.013143] LNetError: 114133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.022915] LNetError: 114133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1822.031806] LNet: Removed LNI 192.168.203.29@tcp [ 1822.044845] LNet: Removed LNI 192.168.203.29@tcp2 [ 1822.088403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.187789] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1822.197698] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1822.758927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1823.047641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.096234] LNet: Removed LNI 192.168.203.29@tcp [ 1824.102851] LNet: Removed LNI 192.168.203.29@tcp2 [ 1824.125859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.216094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1824.250824] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1824.839502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1824.869735] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1825.724520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1825.757725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1825.778981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1825.832097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1826.123318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1827.170213] LNet: Removed LNI 192.168.203.29@tcp [ 1828.203327] LNet: Removed LNI 192.168.203.29@tcp2 [ 1828.273403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.321549] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1828.327405] LNet: Added LNI 192.168.203.29@tcp2 [8/256/0/180] [ 1828.332868] LNet: 114499:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1828.336505] LNet: 114499:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1828.647794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1836.030491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1837.932995] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 16:29:22 (1763155762) [ 1838.144894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.193331] LNet: 115254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.198240] LNet: 115254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1838.223331] LNetError: 115254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1838.239181] LNetError: 115254:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1838.263316] LNet: Removed LNI 192.168.203.29@tcp [ 1839.334985] LNet: Removed LNI 192.168.203.29@tcp2 [ 1839.359630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.727893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.800813] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1840.892690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1840.907283] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1841.787986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1841.817982] LNet: Removed LNI 192.168.203.29@tcp [ 1841.837272] LNet: Removed LNI 10.1.2.3@tcp [ 1841.852103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1841.915812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1841.941954] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1841.959689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1842.836029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1842.887293] LNet: Removed LNI 192.168.203.29@tcp [ 1842.906669] LNet: Removed LNI 10.1.2.3@tcp [ 1849.453375] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 16:29:33 (1763155773) [ 1849.540199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1849.587315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1849.745963] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1849.916615] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1849.930938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1849.955293] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1852.374897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.402747] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1852.970837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1852.985475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1854.779884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.29@tcp [ 1854.821363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1854.895043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.29@tcp -m PUT -e local_timeout -r 1 [ 1854.956682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1855.013283] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1855.015384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1855.234494] LNetError: 116199:0:(lib-move.c:4947:lnet_parse()) 192.168.203.29@tcp, src 192.168.203.29@tcp: Dropping PUT (error -108 looking up sender) [ 1857.312287] LNet: Removed LNI 192.168.203.29@tcp [ 1857.320853] LNet: Removed LNI 10.1.2.3@tcp [ 1857.762034] Key type .llcrypt unregistered [ 1857.763969] Key type ._llcrypt unregistered [ 1865.745901] Key type ._llcrypt registered [ 1865.749250] Key type .llcrypt registered [ 1866.735367] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 16:29:51 (1763155791) [ 1867.040203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1867.052074] alg: No test for adler32 (adler32-zlib) [ 1867.961181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1868.262859] LNet: 117504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1868.912967] Key type .llcrypt unregistered [ 1868.914751] Key type ._llcrypt unregistered [ 1873.918840] Key type ._llcrypt registered [ 1873.920196] Key type .llcrypt registered [ 1874.083706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1874.102604] alg: No test for adler32 (adler32-zlib) [ 1875.034957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1875.079426] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1881.499497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1881.706783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1882.141487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.129@tcp [ 1882.303751] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 1882.432901] LNet: 117941:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1882.485568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1882.520657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1883.040475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1883.341156] LNet: 118046:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1883.402904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1883.427375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1883.992827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.129@tcp [ 1884.326484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1884.361525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1884.855965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1885.100897] LNet: 118257:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1885.108030] LNet: 118257:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 1885.162455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1885.185625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1885.449404] LNet: 118298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.461220] LNetError: 118298:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.507426] LNet: Removed LNI 192.168.203.29@tcp [ 1886.763548] Key type .llcrypt unregistered [ 1886.764741] Key type ._llcrypt unregistered [ 1898.567299] Key type ._llcrypt registered [ 1898.568777] Key type .llcrypt registered [ 1899.548763] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 16:30:24 (1763155824) [ 1899.852169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.860834] alg: No test for adler32 (adler32-zlib) [ 1900.779510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.126991] LNet: 119181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.596341] Key type .llcrypt unregistered [ 1901.597818] Key type ._llcrypt unregistered [ 1905.847981] Key type ._llcrypt registered [ 1905.849261] Key type .llcrypt registered [ 1906.083400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1906.101573] alg: No test for adler32 (adler32-zlib) [ 1907.081160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1907.123521] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1912.937918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.129@tcp [ 1919.649975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1919.682185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1919.700152] LNet: Added LNI 192.168.203.29@tcp1 [8/256/0/180] [ 1920.648039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1921.178955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.129@tcp [ 1921.472814] LNet: 119820:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1921.554492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1921.568428] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.129@tcp [ 1921.600856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1922.163868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1922.422257] LNet: 119925:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1922.501688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1922.558241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1923.165676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.129@tcp [ 1923.490228] LNet: 120030:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1923.595738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1923.638857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1924.320471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1924.671743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1924.701367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1928.245431] LNet: 120211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1928.254275] LNetError: 120211:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.263721] LNet: Removed LNI 192.168.203.29@tcp [ 1928.289632] LNet: Removed LNI 192.168.203.29@tcp1 [ 1928.637869] Key type .llcrypt unregistered [ 1928.639503] Key type ._llcrypt unregistered [ 1939.948179] Key type ._llcrypt registered [ 1939.949479] Key type .llcrypt registered [ 1940.902517] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1760 sec ========== 16:31:05 (1763155865) [ 1941.425251] Key type .llcrypt unregistered [ 1941.428764] Key type ._llcrypt unregistered [ 1949.318744] Key type ._llcrypt registered [ 1949.320438] Key type .llcrypt registered [ 1949.659882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1949.669461] alg: No test for adler32 (adler32-zlib) [ 1950.702391] Lustre: Lustre: Build Version: 2.16.61_2_g297829f [ 1950.951641] LNet: Added LNI 192.168.203.29@tcp [8/256/0/180] [ 1952.624349] Key type lgssc registered [ 1953.656624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.621844] Lustre: Mounted lustre-client [ 2006.997452] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2019.005835] LustreError: 123430:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9d6847068800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2019.026407] LustreError: 123430:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2019.062546] Lustre: Unmounted lustre-client [ 2055.172184] Key type lgssc unregistered [ 2055.389369] LNet: 123913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.396815] LNetError: 123913:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.413934] LNet: Removed LNI 192.168.203.29@tcp [ 2055.953142] Key type .llcrypt unregistered [ 2055.954802] Key type ._llcrypt unregistered