[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 365146147 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002250] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004006] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.006010] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.007005] pid_max: default: 32768 minimum: 301 [ 0.008094] LSM: Security Framework initializing [ 0.009027] Yama: becoming mindful. [ 0.010021] SELinux: Initializing. [ 0.010690] *** VALIDATE selinux *** [ 0.017251] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.021522] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.022082] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.023070] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024061] *** VALIDATE tmpfs *** [ 0.025230] *** VALIDATE proc *** [ 0.026064] *** VALIDATE cgroup *** [ 0.026712] *** VALIDATE cgroup2 *** [ 0.027185] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.028102] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.029003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.030019] Spectre V2 : User space: Vulnerable [ 0.031003] Speculative Store Bypass: Vulnerable [ 0.033771] debug: unmapping init [mem 0xffffffffbb659000-0xffffffffbb660fff] [ 0.035123] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.036411] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.037016] ... version: 2 [ 0.037756] ... bit width: 48 [ 0.038006] ... generic registers: 4 [ 0.038717] ... value mask: 0000ffffffffffff [ 0.039005] ... max period: 00007fffffffffff [ 0.040006] ... fixed-purpose events: 3 [ 0.040695] ... event mask: 000000070000000f [ 0.041877] rcu: Hierarchical SRCU implementation. [ 0.043197] smp: Bringing up secondary CPUs ... [ 0.044393] x86: Booting SMP configuration: [ 0.045015] .... node #0, CPUs: #1 #2 #3 [ 0.047833] smp: Brought up 1 node, 4 CPUs [ 0.048752] smpboot: Max logical packages: 1 [ 0.049006] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.230649] node 0 deferred pages initialised in 180ms [ 0.233609] devtmpfs: initialized [ 0.234178] x86/mm: Memory block size: 128MB [ 0.236282] gcov: version magic: 0x41383552 [ 0.238174] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239076] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.240295] pinctrl core: initialized pinctrl subsystem [ 0.241176] [ 0.241772] ************************************************************* [ 0.242008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.243008] ** ** [ 0.244008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.245008] ** ** [ 0.246006] ** This means that this kernel is built to expose internal ** [ 0.247008] ** IOMMU data structures, which may compromise security on ** [ 0.248008] ** your system. ** [ 0.249006] ** ** [ 0.250006] ** If you see this message and you are not debugging the ** [ 0.251005] ** kernel, report this immediately to your vendor! ** [ 0.252005] ** ** [ 0.253005] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254006] ************************************************************* [ 0.255632] NET: Registered protocol family 16 [ 0.256340] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.257027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.258027] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.259333] cpuidle: using governor menu [ 0.260549] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.263527] PCI: Using configuration type 1 for base access [ 0.265132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.274138] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.275008] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.276127] cryptd: max_cpu_qlen set to 1000 [ 0.278216] ACPI: Added _OSI(Module Device) [ 0.279010] ACPI: Added _OSI(Processor Device) [ 0.279858] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.281011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.285218] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.290317] ACPI: Interpreter enabled [ 0.291039] ACPI: PM: (supports S0 S3 S4 S5) [ 0.291912] ACPI: Using IOAPIC for interrupt routing [ 0.292060] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.295298] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.302240] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.303031] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.305009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.306036] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.308893] acpiphp: Slot [2] registered [ 0.310092] acpiphp: Slot [3] registered [ 0.310883] acpiphp: Slot [4] registered [ 0.311041] acpiphp: Slot [5] registered [ 0.311970] acpiphp: Slot [6] registered [ 0.313075] acpiphp: Slot [7] registered [ 0.313836] acpiphp: Slot [8] registered [ 0.315042] acpiphp: Slot [9] registered [ 0.315791] acpiphp: Slot [10] registered [ 0.316042] acpiphp: Slot [11] registered [ 0.316847] acpiphp: Slot [12] registered [ 0.318077] acpiphp: Slot [13] registered [ 0.319066] acpiphp: Slot [14] registered [ 0.320051] acpiphp: Slot [15] registered [ 0.320921] acpiphp: Slot [16] registered [ 0.321046] acpiphp: Slot [17] registered [ 0.321874] acpiphp: Slot [18] registered [ 0.323046] acpiphp: Slot [19] registered [ 0.323908] acpiphp: Slot [20] registered [ 0.324046] acpiphp: Slot [21] registered [ 0.324894] acpiphp: Slot [22] registered [ 0.326044] acpiphp: Slot [23] registered [ 0.326814] acpiphp: Slot [24] registered [ 0.328045] acpiphp: Slot [25] registered [ 0.328873] acpiphp: Slot [26] registered [ 0.329061] acpiphp: Slot [27] registered [ 0.329912] acpiphp: Slot [28] registered [ 0.331060] acpiphp: Slot [29] registered [ 0.331865] acpiphp: Slot [30] registered [ 0.333050] acpiphp: Slot [31] registered [ 0.333900] PCI host bridge to bus 0000:00 [ 0.334010] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336009] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339012] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340009] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.342011] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.343151] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.345846] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348667] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.354478] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.358590] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.360014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.362009] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.363008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.365588] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.368909] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.372041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.374451] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.378011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.384020] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.387013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.392315] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.396013] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.402017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411017] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.417431] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.421018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.427020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.441029] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.453279] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.455398] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.457409] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.460290] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.461116] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.466137] iommu: Default domain type: Passthrough [ 0.468498] SCSI subsystem initialized [ 0.469110] ACPI: bus type USB registered [ 0.470083] usbcore: registered new interface driver usbfs [ 0.472057] usbcore: registered new interface driver hub [ 0.473059] usbcore: registered new device driver usb [ 0.474104] pps_core: LinuxPPS API ver. 1 registered [ 0.475005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.477023] PTP clock support registered [ 0.478142] EDAC MC: Ver: 3.0.0 [ 0.479419] PCI: Using ACPI for IRQ routing [ 0.480517] NetLabel: Initializing [ 0.481008] NetLabel: domain hash size = 128 [ 0.483011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.485093] NetLabel: unlabeled traffic allowed by default [ 0.487148] vgaarb: loaded [ 0.489237] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.490005] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.494244] clocksource: Switched to clocksource kvm-clock [ 0.580786] VFS: Disk quotas dquot_6.6.0 [ 0.581651] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.582951] *** VALIDATE ramfs *** [ 0.583609] *** VALIDATE hugetlbfs *** [ 0.584426] pnp: PnP ACPI init [ 0.585904] pnp: PnP ACPI: found 6 devices [ 0.599235] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.601070] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.602242] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.603400] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.604703] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.605975] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.607788] NET: Registered protocol family 2 [ 0.609344] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.612393] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.614399] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.618389] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.620984] TCP: Hash tables configured (established 65536 bind 65536) [ 0.623167] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.625602] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.627785] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.629984] NET: Registered protocol family 1 [ 0.632563] RPC: Registered named UNIX socket transport module. [ 0.634217] RPC: Registered udp transport module. [ 0.635404] RPC: Registered tcp transport module. [ 0.636347] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.637945] NET: Registered protocol family 44 [ 0.638872] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.640035] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.641174] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.642411] PCI: CLS 0 bytes, default 64 [ 0.643273] Unpacking initramfs... [ 1.937732] debug: unmapping init [mem 0xffff88c5fcc64000-0xffff88c5fffcffff] [ 1.940930] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 1.942148] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 1.943697] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.349501] Initialise system trusted keyrings [ 2.350802] Key type blacklist registered [ 2.352738] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.360749] zbud: loaded [ 2.363511] *** VALIDATE nfs *** [ 2.364306] *** VALIDATE nfs4 *** [ 2.366031] pstore: using deflate compression [ 2.368181] Platform Keyring initialized [ 2.446740] NET: Registered protocol family 38 [ 2.447617] Key type asymmetric registered [ 2.448449] Asymmetric key parser 'x509' registered [ 2.450034] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.451583] io scheduler mq-deadline registered [ 2.452586] io scheduler kyber registered [ 2.453532] io scheduler bfq registered [ 2.454515] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.455937] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.457315] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.458748] ACPI: Power Button [PWRF] [ 2.544278] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.630305] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.719541] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.747166] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.779322] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.792321] Non-volatile memory driver v1.3 [ 2.793810] Linux agpgart interface v0.103 [ 2.819847] virtio_blk virtio1: [vda] 133240 512-byte logical blocks (68.2 MB/65.1 MiB) [ 2.822151] vda: detected capacity change from 0 to 68218880 [ 2.893868] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.895562] vdb: detected capacity change from 0 to 1073741824 [ 2.900469] libphy: Fixed MDIO Bus: probed [ 2.904877] usbcore: registered new interface driver usbserial_generic [ 2.906845] usbserial: USB Serial support registered for generic [ 2.908788] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.912774] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.914188] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.916363] mousedev: PS/2 mouse device common for all mice [ 2.918638] rtc_cmos 00:05: RTC can wake from S4 [ 2.921734] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.921963] rtc_cmos 00:05: registered as rtc0 [ 2.925846] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.928620] intel_pstate: CPU model not supported [ 2.931437] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.932702] hid: raw HID events driver (C) Jiri Kosina [ 2.935938] usbcore: registered new interface driver usbhid [ 2.937301] usbhid: USB HID core driver [ 2.938747] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.939430] drop_monitor: Initializing network drop monitor service [ 2.943027] Initializing XFRM netlink socket [ 2.944312] NET: Registered protocol family 10 [ 2.946375] Segment Routing with IPv6 [ 2.947153] NET: Registered protocol family 17 [ 2.948407] mpls_gso: MPLS GSO support [ 2.954191] RAS: Correctable Errors collector initialized. [ 2.956133] AVX version of gcm_enc/dec engaged. [ 2.957189] AES CTR mode by8 optimization enabled [ 3.023269] sched_clock: Marking stable (3023245418, 0)->(3711065648, -687820230) [ 3.025424] registered taskstats version 1 [ 3.026672] Loading compiled-in X.509 certificates [ 3.028738] zswap: loaded using pool lzo/zbud [ 3.047909] Key type big_key registered [ 3.057430] Key type encrypted registered [ 3.058303] ima: No TPM chip found, activating TPM-bypass! [ 3.059562] ima: Allocated hash algorithm: sha1 [ 3.061331] ima: No architecture policies found [ 3.062662] evm: Initialising EVM extended attributes: [ 3.063574] evm: security.selinux [ 3.064199] evm: security.ima [ 3.064792] evm: security.capability [ 3.065565] evm: HMAC attrs: 0x1 [ 3.067301] rtc_cmos 00:05: setting system clock to 2025-09-25 18:06:57 UTC (1758823617) [ 3.071566] debug: unmapping init [mem 0xffffffffbc603000-0xffffffffbc7fffff] [ 3.073051] debug: unmapping init [mem 0xffffffffbb382000-0xffffffffbb658fff] [ 3.081078] Write protecting the kernel read-only data: 28672k [ 3.083086] debug: unmapping init [mem 0xffffffffb9a03000-0xffffffffb9bfffff] [ 3.084722] debug: unmapping init [mem 0xffffffffba314000-0xffffffffba3fffff] [ 3.113544] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.118395] systemd[1]: Detected virtualization kvm. [ 3.119391] systemd[1]: Detected architecture x86-64. [ 3.120680] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.145756] systemd[1]: No hostname configured. [ 3.147117] systemd[1]: Set hostname to . [ 3.148805] random: systemd: uninitialized urandom read (16 bytes read) [ 3.150740] systemd[1]: Initializing machine ID from random generator. [ 3.185315] random: ln: uninitialized urandom read (6 bytes read) [ 3.264245] random: systemd: uninitialized urandom read (16 bytes read) [ 3.266896] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.271370] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.273854] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Static Device Nodes in /dev. Starting dracut cmdline hook... [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.838297] device-mapper: uevent: version 1.0.3 [ 3.839965] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.538046] virtio_net virtio0 ens2: renamed from eth0 [ 4.707788] scsi host0: ata_piix [ 4.728722] scsi host1: ata_piix [ 4.730111] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.732059] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 5.932739] random: fast init done [ 9.597490] random: crng init done [ 9.603049] random: 7 urandom warning(s) missed due to ratelimiting [ 10.220761] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.234748] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.227857] printk: systemd: 22 output lines suppressed due to ratelimiting [ 13.670503] SELinux: Disabled at runtime. [ 13.765399] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 13.772895] systemd[1]: Detected virtualization kvm. [ 13.774751] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.896272] hrtimer: interrupt took 2287597 ns [ 14.746270] systemd[1]: initrd-switch-root.service: Succeeded. [ 14.751822] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 14.758626] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 14.762151] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 14.765239] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 14.782564] systemd[1]: Starting Journal Service... Starting Journal Service... [ 14.812901] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 14.977959] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 15.459957] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.929904] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.930919] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 16.137383] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 16.187836] EDAC sbridge: Ver: 1.1.2 [ 17.860378] Key type dns_resolver registered [ 18.206731] NFS: Registering the id_resolver key type [ 18.208469] Key type id_resolver registered [ 18.209502] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg351-client login: [ 47.746569] libcfs: loading out-of-tree module taints kernel. [ 47.777366] Key type ._llcrypt registered [ 47.778295] Key type .llcrypt registered [ 48.043592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 48.048051] alg: No test for adler32 (adler32-zlib) [ 48.998484] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 49.255068] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 50.863135] Key type lgssc registered [ 51.384989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.803446] Lustre: Mounted lustre-client [ 98.159823] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 110.706364] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing check_logdir /tmp/testlogs/ [ 112.229654] Lustre: DEBUG MARKER: oleg351-client.virtnet: executing yml_node [ 113.593186] Lustre: DEBUG MARKER: Client: 2.16.58.75 [ 114.375429] Lustre: DEBUG MARKER: MDS: 2.16.58.75 [ 115.154164] Lustre: DEBUG MARKER: OSS: 2.16.58.75 [ 115.672684] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Sep 25 14:08:49 EDT 2025 [ 120.490465] LustreError: 7902:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff88c64ad36800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 120.497447] LustreError: 7902:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 120.508092] Lustre: Unmounted lustre-client [ 146.209275] Key type lgssc unregistered [ 146.332534] LNet: 8349:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 146.335172] LNetError: 8349:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 146.341396] LNet: Removed LNI 192.168.203.51@tcp [ 146.616114] Key type .llcrypt unregistered [ 146.617303] Key type ._llcrypt unregistered [ 154.894422] Key type ._llcrypt registered [ 154.895612] Key type .llcrypt registered [ 154.965760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 154.972472] alg: No test for adler32 (adler32-zlib) [ 155.772246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 155.795202] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 155.803615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 157.174765] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 157.208968] LNet: 9212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 157.212389] LNetError: 9212:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 157.219521] LNet: Removed LNI 192.168.203.51@tcp [ 157.470449] Key type .llcrypt unregistered [ 157.472228] Key type ._llcrypt unregistered [ 159.610133] Key type ._llcrypt registered [ 159.611695] Key type .llcrypt registered [ 160.141464] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 14:09:33 (1758823773) [ 160.267851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 160.275375] alg: No test for adler32 (adler32-zlib) [ 161.089931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 161.123389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 161.133110] LNet: 9626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 161.143373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 161.165448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 161.176751] LNet: 9633:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 163.674456] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 14:09:37 (1758823777) [ 163.709277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 163.719797] LNet: 10204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 163.733412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 163.771164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 166.022983] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 14:09:39 (1758823779) [ 166.062757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 166.075361] LNet: 10788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 166.086884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 166.125155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 168.501192] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 14:09:42 (1758823782) [ 168.538659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 168.550278] LNet: 11372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 168.561427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 168.597583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 171.034461] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 14:09:44 (1758823784) [ 171.068444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 171.090302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 171.134053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 171.314335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 173.830648] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 14:09:47 (1758823787) [ 173.864634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 173.875205] LNet: 12544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 173.877209] LNet: 12544:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 173.887230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 173.925990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 174.090431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 174.265364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 176.725145] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 14:09:50 (1758823790) [ 176.759472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 176.780727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 176.821139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 179.280944] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 14:09:53 (1758823793) [ 179.309716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 179.330039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 179.363640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 179.378983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 179.548820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 179.563521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 179.732219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 179.750828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 179.909134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 179.926907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 180.069866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 180.086337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 180.239274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 180.254034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 180.424384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 180.442151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 180.597528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 180.611351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 180.779441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 180.794204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 180.941669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 180.958647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 181.136729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 181.153389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 181.316226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 181.333963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 181.500816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 181.518831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 184.038856] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 14:09:57 (1758823797) [ 184.076734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.086891] LNet: 14406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 184.090381] LNet: 14406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 184.101284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.132776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 184.147133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 186.631309] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 14:10:00 (1758823800) [ 186.663466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 186.681187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 186.713382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 186.728788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 189.084218] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 14:10:03 (1758823803) [ 189.121146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 189.139853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 189.176754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 189.194190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 191.590116] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 14:10:05 (1758823805) [ 191.622143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 191.642167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 191.677565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 191.691985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 194.039343] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 14:10:07 (1758823807) [ 194.074892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 194.094397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 194.137825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 194.155939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 196.529751] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 14:10:10 (1758823810) [ 196.561446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.584420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.620671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 196.640848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 199.017943] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 14:10:12 (1758823812) [ 199.057591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.077871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.122285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 199.302188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 201.716614] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 14:10:15 (1758823815) [ 201.757507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 201.770836] LNet: 18511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 201.773803] LNet: 18511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 201.785348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 201.875836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 201.969127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 204.312763] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 14:10:18 (1758823818) [ 204.336934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 204.354627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 204.407995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 204.496687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 206.672860] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 14:10:20 (1758823820) [ 206.706688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 206.730365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 206.892926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 206.998994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 209.255774] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 14:10:23 (1758823823) [ 209.287844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 209.306652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 209.338130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 209.357858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 211.542825] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 14:10:25 (1758823825) [ 211.573928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 211.592606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 211.630674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 211.646961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 213.907655] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 14:10:27 (1758823827) [ 213.947239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 213.970014] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 214.003988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 216.164485] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 14:10:30 (1758823830) [ 216.193758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 216.213792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 216.253787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 218.465840] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 14:10:32 (1758823832) [ 218.498853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.514652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.549199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 218.703770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 220.896908] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 14:10:34 (1758823834) [ 220.926153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 220.941069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 220.974418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 220.988428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 223.156706] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 14:10:37 (1758823837) [ 223.193071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 223.211862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 223.248351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 223.261344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 225.352146] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 14:10:39 (1758823839) [ 225.379255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 225.396319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 225.430342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 225.448336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 227.533117] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 14:10:41 (1758823841) [ 227.562883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.581574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.615598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 227.632500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 229.752403] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 14:10:43 (1758823843) [ 229.782368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 229.799210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 229.822627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 229.848192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 229.856408] LNetError: 25711:0:(peer.c:2209:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 229.871117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 229.887104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 231.865970] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 14:10:45 (1758823845) [ 231.890491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 231.907079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.941564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3nJYJlah4o [ 231.973205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3nJYJlah4o [ 231.996824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.3nJYJlah4o [ 233.956855] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 14:10:47 (1758823847) [ 233.980829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 233.991375] LNet: 26875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.995126] LNet: 26875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 234.005347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.036075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 234.048787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 234.067536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 236.041311] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 14:10:50 (1758823850) [ 236.074562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 236.089799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 236.117568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 237.069551] LNet: Message forwarding will be enabled in 2s [ 240.159750] LNet: Message forwarding enabled [ 240.225107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 240.234382] LNet: Message forwarding will stop in 2s to 6s [ 244.427881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 245.017330] LNet: Message forwarding disabled [ 245.543530] LNet: Message forwarding will be enabled in 2s [ 249.119095] LNet: Message forwarding enabled [ 249.664710] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 14:11:03 (1758823863) [ 249.793171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.614569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.693921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 255.019092] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 14:11:08 (1758823868) [ 255.289153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 255.342259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 255.394248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 255.424348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 255.457076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 255.492751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 255.526187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 255.555490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 255.588583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 255.619834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 255.648522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 255.678401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 255.701554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 255.737211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 255.776544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 255.808994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 255.839933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 255.867523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 255.897826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 255.930012] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 255.979749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 255.994830] LNetError: 28778:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 260.482939] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 14:11:13 (1758823873) [ 260.562686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.598525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.669195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 265.333779] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 14:11:18 (1758823878) [ 266.573677] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg351-server [ 267.990119] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 14:11:21 (1758823881) [ 268.082766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.121258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.176894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 268.202449] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 269.734363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[52-56]@tcp [ 269.745626] LNet: 30490:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 269.775166] LNetError: 30287:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.52@tcp has gone from up to down [ 272.863378] LNetError: 30492:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.55@tcp has gone from up to down [ 274.981413] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 14:11:28 (1758823888) [ 276.534944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg351-server [ 277.815036] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 14:11:31 (1758823891) [ 277.898556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 277.921292] LNetError: 31421:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 277.947658] LNet: Removed LNI 192.168.203.51@tcp [ 277.979543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 278.022328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 278.038831] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 278.470934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[52-57/2]@tcp [ 278.485524] LNet: 31539:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 278.489652] LNet: 31539:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 278.497709] LNetError: 31433:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.52@tcp has gone from up to down [ 278.503691] LNetError: 31433:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 278.519604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[52-57/2]@tcp [ 282.073147] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 14:11:35 (1758823895) [ 282.107447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.119830] LNetError: 32124:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 282.128156] LNet: Removed LNI 192.168.203.51@tcp [ 282.139280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.185547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 282.214680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.237649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.280357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 282.307100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.324501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.351533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 282.393236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.412340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.449192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 282.488883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.507629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.538514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 282.577580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.596807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.632834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 282.670413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.690590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.719377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 282.755671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.774519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.807852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 282.848850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.869028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.904802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 282.941235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.960235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.992489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 283.033651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.051443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.077211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 283.102347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.121499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.153203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 285.943558] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 14:11:39 (1758823899) [ 286.757768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg351-server [ 287.476498] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 14:11:41 (1758823901) [ 288.212589] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg351-server [ 288.939631] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 14:11:42 (1758823902) [ 289.341648] Key type .llcrypt unregistered [ 289.343320] Key type ._llcrypt unregistered [ 291.678499] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 291.773811] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 291.777190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 291.780093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 293.985357] Key type ._llcrypt registered [ 293.986774] Key type .llcrypt registered [ 294.057554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 294.064599] alg: No test for adler32 (adler32-zlib) [ 294.887434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.919578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 294.934928] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 295.232981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 295.244379] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 296.185743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 296.197723] LNet: Removed LNI 10.1.2.3@tcp [ 296.261402] LNet: 34241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.265809] LNetError: 34241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.275257] LNet: Removed LNI 192.168.203.51@tcp [ 296.594617] Key type .llcrypt unregistered [ 296.596378] Key type ._llcrypt unregistered [ 301.184965] Key type ._llcrypt registered [ 301.186373] Key type .llcrypt registered [ 301.743734] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 14:11:55 (1758823915) [ 301.964517] Key type .llcrypt unregistered [ 301.966349] Key type ._llcrypt unregistered [ 304.204384] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 304.251119] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 304.253912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 304.256884] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 306.566050] Key type ._llcrypt registered [ 306.569947] Key type .llcrypt registered [ 306.665811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 306.675340] alg: No test for adler32 (adler32-zlib) [ 307.509310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.550347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 307.569820] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 307.950831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 307.969306] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 309.170664] LNet: 35720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.175236] LNetError: 35720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.178586] LNet: Removed LNI 192.168.203.51@tcp [ 309.191252] LNet: Removed LNI 10.1.2.3@tcp [ 309.572685] Key type .llcrypt unregistered [ 309.574477] Key type ._llcrypt unregistered [ 314.383328] Key type ._llcrypt registered [ 314.385034] Key type .llcrypt registered [ 315.079793] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 14:12:08 (1758823928) [ 315.347350] Key type .llcrypt unregistered [ 315.350547] Key type ._llcrypt unregistered [ 317.927230] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 317.959677] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 317.961824] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 317.964682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 320.269805] Key type ._llcrypt registered [ 320.271382] Key type .llcrypt registered [ 320.380070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 320.387479] alg: No test for adler32 (adler32-zlib) [ 321.238653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.267068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 321.284941] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 322.399964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 322.419388] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 323.618814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 323.633173] LNet: Removed LNI 10.1.2.3@tcp [ 323.642589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 323.662202] LNet: Removed LNI 10.1.2.31@tcp [ 323.674890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 323.693963] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 324.065902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 324.079318] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 324.614359] LNet: 37619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.619433] LNetError: 37619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 324.622644] LNet: Removed LNI 10.1.2.3@tcp [ 324.629583] LNet: Removed LNI 10.1.2.31@tcp [ 324.993569] Key type .llcrypt unregistered [ 324.995187] Key type ._llcrypt unregistered [ 329.966229] Key type ._llcrypt registered [ 329.967809] Key type .llcrypt registered [ 330.587931] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 14:12:24 (1758823944) [ 330.743020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 330.750749] alg: No test for adler32 (adler32-zlib) [ 331.574879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.604991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 331.619955] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 332.040431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.058866] LNet: 38563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.063490] LNetError: 38563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.073062] LNet: Removed LNI 192.168.203.51@tcp [ 332.087258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.120296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.51@tcp [ 332.134582] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 332.503838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.51@tcp [ 332.529534] LNet: Removed LNI 192.168.203.51@tcp [ 332.558515] LNet: 38695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.903495] Key type .llcrypt unregistered [ 332.904940] Key type ._llcrypt unregistered [ 337.624430] Key type ._llcrypt registered [ 337.626184] Key type .llcrypt registered [ 338.285825] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 14:12:31 (1758823951) [ 339.159747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg351-server [ 339.995467] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 14:12:33 (1758823953) [ 340.198528] Key type .llcrypt unregistered [ 340.200377] Key type ._llcrypt unregistered [ 342.427469] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 342.486688] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 342.489117] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 342.491697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 344.737693] Key type ._llcrypt registered [ 344.738894] Key type .llcrypt registered [ 344.812828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 344.818916] alg: No test for adler32 (adler32-zlib) [ 345.637483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.669445] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 345.999899] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 346.828170] LNet: 40517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 346.832311] LNetError: 40517:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 346.835875] LNet: Removed LNI 192.168.203.51@tcp1 [ 346.844423] LNet: Removed LNI 10.1.2.3@tcp2 [ 347.110627] Key type .llcrypt unregistered [ 347.111870] Key type ._llcrypt unregistered [ 351.126463] Key type ._llcrypt registered [ 351.128105] Key type .llcrypt registered [ 351.671404] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 14:12:45 (1758823965) [ 351.807256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 351.814669] alg: No test for adler32 (adler32-zlib) [ 352.625367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 352.650127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 352.663102] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 352.985492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 352.994655] LNet: 41474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 352.997674] LNetError: 41474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 353.006478] LNet: Removed LNI 192.168.203.51@tcp [ 353.015456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.062647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 353.073733] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 355.886351] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 14:12:49 (1758823969) [ 355.932383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.942076] LNet: 42066:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.946331] LNetError: 42066:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.958157] LNet: Removed LNI 192.168.203.51@tcp [ 355.974956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.998574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 356.008829] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 356.388176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.51 [ 356.399316] LNet: Removed LNI 192.168.203.51@tcp [ 358.850520] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 14:12:52 (1758823972) [ 358.887650] LNet: 42762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.891962] LNetError: 42762:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 358.901576] LNet: Removed LNI 192.168.203.51@tcp [ 359.186696] Key type .llcrypt unregistered [ 359.188228] Key type ._llcrypt unregistered [ 361.379737] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 361.503853] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 361.506068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 361.508319] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 363.788237] Key type ._llcrypt registered [ 363.789820] Key type .llcrypt registered [ 363.852889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 363.860317] alg: No test for adler32 (adler32-zlib) [ 364.835058] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 367.170823] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 14:13:01 (1758823981) [ 367.201560] LNet: 43661:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 367.205637] LNetError: 43661:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 367.213871] LNet: Removed LNI 10.1.2.3@tcp [ 367.505697] Key type .llcrypt unregistered [ 367.507774] Key type ._llcrypt unregistered [ 369.595829] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 369.699197] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 369.701725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 369.704202] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 371.989249] Key type ._llcrypt registered [ 371.990936] Key type .llcrypt registered [ 372.060505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 372.068057] alg: No test for adler32 (adler32-zlib) [ 372.906681] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 375.593047] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 14:13:09 (1758823989) [ 375.639605] LNet: 44549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.645341] LNetError: 44549:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 375.655643] LNet: Removed LNI 10.1.2.3@tcp [ 375.946873] Key type .llcrypt unregistered [ 375.948588] Key type ._llcrypt unregistered [ 378.282626] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 378.401632] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 378.403770] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 378.406140] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 380.713353] Key type ._llcrypt registered [ 380.714895] Key type .llcrypt registered [ 380.775684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 380.782448] alg: No test for adler32 (adler32-zlib) [ 381.627175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 383.860290] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 14:13:17 (1758823997) [ 383.901313] LNet: 45438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.905869] LNetError: 45438:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 383.916808] LNet: Removed LNI 10.1.2.3@tcp [ 384.174671] Key type .llcrypt unregistered [ 384.176358] Key type ._llcrypt unregistered [ 386.236276] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 386.360682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 386.362954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 386.365364] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 388.652808] Key type ._llcrypt registered [ 388.654228] Key type .llcrypt registered [ 388.722476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 388.729074] alg: No test for adler32 (adler32-zlib) [ 389.549175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.586901] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 392.953571] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 14:13:26 (1758824006) [ 393.127509] LNet: 46665:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.129932] LNetError: 46665:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 393.140836] LNet: Removed LNI 10.1.2.3@tcp [ 393.437607] Key type .llcrypt unregistered [ 393.439043] Key type ._llcrypt unregistered [ 395.771359] Key type ._llcrypt registered [ 395.772948] Key type .llcrypt registered [ 395.845397] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 395.853428] alg: No test for adler32 (adler32-zlib) [ 396.661360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 396.675092] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 399.931818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 400.195291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 400.213335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 400.233746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 400.244743] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 400.373789] LNet: 47089:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 400.405259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 400.420877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 400.439072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 400.667922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 400.685364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 400.702967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 400.872158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 400.885807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 400.901124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 401.109575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 401.126270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 401.144854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 401.155707] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 401.159150] LNet: Skipped 1 previous similar message [ 401.286784] LNet: 47283:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 401.290328] LNet: 47283:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 401.323747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 401.346701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 401.363219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 401.570362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 401.586272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 401.603978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 401.782497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 401.799380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 401.816595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 402.022166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 402.040043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 402.058190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 402.237238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 402.255769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 402.272464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 402.480353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 402.497488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 402.514484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 402.524441] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 402.526834] LNet: Skipped 2 previous similar messages [ 402.665239] LNet: 47575:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 402.668679] LNet: 47575:0:(api-ni.c:339:recovery_interval_set()) Skipped 2 previous similar messages [ 402.702493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 402.721812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 402.740847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 402.861285] LNet: 47611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.865409] LNetError: 47611:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 402.876769] LNet: Removed LNI 192.168.203.51@tcp [ 403.044497] Key type .llcrypt unregistered [ 403.046227] Key type ._llcrypt unregistered [ 408.912746] Key type ._llcrypt registered [ 408.914479] Key type .llcrypt registered [ 409.462918] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 14:13:43 (1758824023) [ 409.780689] Key type .llcrypt unregistered [ 409.781846] Key type ._llcrypt unregistered [ 411.971838] Key type ._llcrypt registered [ 411.973531] Key type .llcrypt registered [ 412.053042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 412.060505] alg: No test for adler32 (adler32-zlib) [ 412.881275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 412.899078] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 416.116863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 419.712417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.728977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 419.739768] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 420.355379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 420.372520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 1 [ 420.394368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 420.410827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 1 [ 420.431101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_interrupt -r 1 [ 420.451956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 1 [ 420.472199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_interrupt -r 1 [ 420.489813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 1 [ 420.508517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 420.519520] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 420.522474] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 420.663633] LNet: 49056:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 420.700502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 420.718046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 420.734414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 420.956944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 420.974089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 1 [ 420.990299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 421.007255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 1 [ 421.027291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_dropped -r 1 [ 421.046101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 1 [ 421.063165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_dropped -r 1 [ 421.080301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 1 [ 421.098202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 421.109339] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 421.112049] LNet: Skipped 2 previous similar messages [ 421.251617] LNet: 49165:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 421.288193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.307483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 421.326684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 421.559768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 421.577663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 1 [ 421.597488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 421.615403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 1 [ 421.635763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_aborted -r 1 [ 421.657650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 1 [ 421.681807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_aborted -r 1 [ 421.706488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 1 [ 421.725349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 421.924204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 421.940799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 421.958424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 422.206668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 422.226703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 1 [ 422.247510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 422.266468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 1 [ 422.286414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_no_route -r 1 [ 422.304777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 1 [ 422.323141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_no_route -r 1 [ 422.343604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 1 [ 422.364222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 422.376796] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 422.379647] LNet: Skipped 6 previous similar messages [ 422.535576] LNet: 49383:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 422.539266] LNet: 49383:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 422.574995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 422.591572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 422.609955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 422.853361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 422.870182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 1 [ 422.888991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 422.907523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 1 [ 422.928615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_timeout -r 1 [ 422.949262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 1 [ 422.968402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_timeout -r 1 [ 422.988907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 1 [ 423.010464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 423.213171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 423.229414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 423.248085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 423.492932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 423.515072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_error -r 1 [ 423.534575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 423.554926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_error -r 1 [ 423.575374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_error -r 1 [ 423.593574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_error -r 1 [ 423.612464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_error -r 1 [ 423.632781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_error -r 1 [ 423.649306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 423.825218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 423.844645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 423.861861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 426.094966] LNet: 49672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 426.097727] LNetError: 49672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 426.101311] LNet: Removed LNI 192.168.203.51@tcp [ 427.109622] LNet: Removed LNI 192.168.203.51@tcp1 [ 427.278870] Key type .llcrypt unregistered [ 427.279848] Key type ._llcrypt unregistered [ 433.322913] Key type ._llcrypt registered [ 433.324511] Key type .llcrypt registered [ 433.904319] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 14:14:07 (1758824047) [ 434.245250] Key type .llcrypt unregistered [ 434.246515] Key type ._llcrypt unregistered [ 436.405532] Key type ._llcrypt registered [ 436.406779] Key type .llcrypt registered [ 436.505712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 436.514781] alg: No test for adler32 (adler32-zlib) [ 437.338484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 437.353636] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 440.645566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 440.956304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 440.974409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 440.994146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 441.004742] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 441.160955] LNet: 50906:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 441.197177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 441.215481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 441.237314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 441.494285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 441.512218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 441.532994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 441.544785] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 441.698092] LNet: 51003:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 441.734864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 441.751699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 441.767253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 442.010241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 442.029515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 442.047351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 442.226685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 442.243085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 442.261641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 442.377097] LNet: 51136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 442.380323] LNetError: 51136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 442.389448] LNet: Removed LNI 192.168.203.51@tcp [ 442.578761] Key type .llcrypt unregistered [ 442.579989] Key type ._llcrypt unregistered [ 448.695650] Key type ._llcrypt registered [ 448.697201] Key type .llcrypt registered [ 449.296409] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 14:14:23 (1758824063) [ 449.651432] Key type .llcrypt unregistered [ 449.653091] Key type ._llcrypt unregistered [ 451.915328] Key type ._llcrypt registered [ 451.916904] Key type .llcrypt registered [ 451.998027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 452.004929] alg: No test for adler32 (adler32-zlib) [ 452.829924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 452.845811] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 456.153023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 459.378232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.394145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 459.405840] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 460.006420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 460.024643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 1 [ 460.043231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 460.063144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 1 [ 460.082203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_dropped -r 1 [ 460.102727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 1 [ 460.121716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_dropped -r 1 [ 460.141327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 1 [ 460.160155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 460.170858] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 460.174505] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 460.319445] LNet: 52582:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 460.357365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 460.373951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 460.391785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 460.411856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 460.642747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 460.661136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_error -r 1 [ 460.680459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 460.699592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_error -r 1 [ 460.719163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_error -r 1 [ 460.739512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_error -r 1 [ 460.759650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_error -r 1 [ 460.781334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_error -r 1 [ 460.801183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 460.812793] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 460.815420] LNet: Skipped 2 previous similar messages [ 460.965531] LNet: 52693:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 461.005023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 461.023920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 461.042552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 461.061884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 461.311882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 461.331363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_timeout -r 1 [ 461.351798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 461.372594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_timeout -r 1 [ 461.391895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_timeout -r 1 [ 461.413875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_timeout -r 1 [ 461.432727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_timeout -r 1 [ 461.455152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_timeout -r 1 [ 461.473872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.151@tcp [ 461.650136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 461.668095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 461.685697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 461.704692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 464.136316] LNet: 52876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.140563] LNetError: 52876:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.143837] LNet: Removed LNI 192.168.203.51@tcp [ 464.157460] LNet: Removed LNI 192.168.203.51@tcp1 [ 464.347609] Key type .llcrypt unregistered [ 464.348886] Key type ._llcrypt unregistered [ 470.644601] Key type ._llcrypt registered [ 470.645694] Key type .llcrypt registered [ 471.195562] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 14:14:45 (1758824085) [ 471.406620] Key type .llcrypt unregistered [ 471.408926] Key type ._llcrypt unregistered [ 473.560708] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 473.672116] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 473.674852] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 473.677203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 475.879890] Key type ._llcrypt registered [ 475.880788] Key type .llcrypt registered [ 475.945563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 475.952078] alg: No test for adler32 (adler32-zlib) [ 476.769021] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 476.787771] LNet: 53960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.791449] LNetError: 53960:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 476.800457] LNet: Removed LNI 192.168.203.51@tcp [ 477.088676] Key type .llcrypt unregistered [ 477.090073] Key type ._llcrypt unregistered [ 479.192092] Key type ._llcrypt registered [ 479.193499] Key type .llcrypt registered [ 479.266430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 479.273395] alg: No test for adler32 (adler32-zlib) [ 480.094691] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 480.096773] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 480.113264] LNet: 54229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 480.116722] LNetError: 54229:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 480.119876] LNet: Removed LNI 192.168.203.51@tcp [ 480.133405] LNet: Removed LNI 10.1.2.3@tcp1 [ 480.439481] Key type .llcrypt unregistered [ 480.441065] Key type ._llcrypt unregistered [ 482.624398] Key type ._llcrypt registered [ 482.625888] Key type .llcrypt registered [ 482.684720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 482.690815] alg: No test for adler32 (adler32-zlib) [ 483.510635] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 483.513093] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 483.531676] LNet: 54497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.535488] LNetError: 54497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 483.537880] LNet: Removed LNI 192.168.203.51@tcp [ 483.544568] LNet: Removed LNI 10.1.2.3@tcp [ 483.839966] Key type .llcrypt unregistered [ 483.841488] Key type ._llcrypt unregistered [ 485.969654] Key type ._llcrypt registered [ 485.971131] Key type .llcrypt registered [ 486.039527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 486.046677] alg: No test for adler32 (adler32-zlib) [ 486.871343] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 486.873657] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 486.890632] LNet: 54767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.894860] LNetError: 54767:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 486.897763] LNet: Removed LNI 192.168.203.51@tcp [ 486.905476] LNet: Removed LNI 10.1.2.3@tcp [ 487.214716] Key type .llcrypt unregistered [ 487.216221] Key type ._llcrypt unregistered [ 489.534086] Key type ._llcrypt registered [ 489.535459] Key type .llcrypt registered [ 489.609847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 489.617067] alg: No test for adler32 (adler32-zlib) [ 490.439534] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 490.441406] LNetError: 55026:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 490.444538] LNetError: Error -100 starting up LNI tcp [ 490.452742] LNet: Removed LNI 192.168.203.51@tcp [ 490.733585] Key type .llcrypt unregistered [ 490.734687] Key type ._llcrypt unregistered [ 494.834642] Key type ._llcrypt registered [ 494.835881] Key type .llcrypt registered [ 495.363743] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 14:15:09 (1758824109) [ 495.694332] Key type .llcrypt unregistered [ 495.695914] Key type ._llcrypt unregistered [ 497.897477] Key type ._llcrypt registered [ 497.898856] Key type .llcrypt registered [ 497.976265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 497.984420] alg: No test for adler32 (adler32-zlib) [ 498.801775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 498.813504] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 501.817149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 502.053913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 502.071743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 502.088557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 502.098346] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 502.226781] LNet: 56264:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 502.256316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 502.270837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 502.377097] LNet: 56298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 502.381398] LNetError: 56298:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 502.390460] LNet: Removed LNI 192.168.203.51@tcp [ 502.554411] Key type .llcrypt unregistered [ 502.555715] Key type ._llcrypt unregistered [ 508.648724] Key type ._llcrypt registered [ 508.650198] Key type .llcrypt registered [ 508.722133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 508.729141] alg: No test for adler32 (adler32-zlib) [ 509.541961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 509.555450] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 512.541573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 515.481990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.496986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 515.507626] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 516.031982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 516.049624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e network_timeout -r 1 [ 516.067414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 516.084621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e network_timeout -r 1 [ 516.102740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e network_timeout -r 1 [ 516.120819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e network_timeout -r 1 [ 516.139640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e network_timeout -r 1 [ 516.158934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e network_timeout -r 1 [ 516.177198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 516.186896] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 516.317847] LNet: 57156:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 516.349639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 516.364860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 518.524364] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 518.630330] LNet: 57224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.633260] LNetError: 57224:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 518.637292] LNet: Removed LNI 192.168.203.51@tcp [ 518.646554] LNet: Removed LNI 192.168.203.51@tcp1 [ 518.821508] Key type .llcrypt unregistered [ 518.823058] Key type ._llcrypt unregistered [ 524.634173] Key type ._llcrypt registered [ 524.635710] Key type .llcrypt registered [ 525.170859] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 14:15:39 (1758824139) [ 525.302019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 525.308030] alg: No test for adler32 (adler32-zlib) [ 526.120052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.144697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 526.157388] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 526.423636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 526.432817] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 526.747701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 526.778215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 526.798399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 526.816511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 526.834919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 526.851582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 526.867167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 526.877380] LNetError: 58313:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 527.135444] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 527.135741] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 529.183362] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 530.052950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 533.279301] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 534.262530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 534.279709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 534.310661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.322196] LNet: 58430:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 534.325741] LNetError: 58430:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.329607] LNet: Removed LNI 192.168.203.51@tcp [ 534.335621] LNet: Removed LNI 192.168.203.51@tcp1 [ 534.346771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.373308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 534.387422] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 534.701035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 534.711780] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 535.077973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 535.098905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 535.125449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 535.148467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 535.167849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 535.190075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 535.207893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 535.226553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.51@tcp [ 535.237578] LNetError: 58696:0:(api-ni.c:4316:lnet_ni_set_healthv()) manually adding local NI 192.168.203.51@tcp to recovery [ 535.391568] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 538.432255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 541.535435] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 541.538345] LNet: Skipped 1 previous similar message [ 545.770121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 545.787076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 545.804720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 548.394763] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 14:16:02 (1758824162) [ 548.423824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.437158] LNet: 59406:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 548.440571] LNetError: 59406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 548.443638] LNetError: 59406:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 548.446585] LNet: Removed LNI 192.168.203.51@tcp [ 548.452485] LNet: Removed LNI 192.168.203.51@tcp1 [ 548.466492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 548.491731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 548.503434] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 548.781867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 548.791491] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 549.135886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 549.164987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 549.179829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 549.195986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 549.213225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 549.229720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 549.244919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 0 [ 549.288924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 549.535138] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.51@tcp [ 550.559373] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 550.564168] LNet: Skipped 1 previous similar message [ 565.853293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 565.919408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 565.933917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 565.949286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 500 [ 565.993102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 571.188290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 571.214604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 571.224885] LNet: 59906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.228808] LNetError: 59906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.232499] LNetError: 59906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 571.236026] LNet: Removed LNI 192.168.203.51@tcp [ 571.241447] LNet: Removed LNI 192.168.203.51@tcp1 [ 571.251142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 571.278340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 571.291256] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 571.577900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 571.587828] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 571.921423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 571.940818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 571.968728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 571.987145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 572.003424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 572.019908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 572.037412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 572.054606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.51@tcp --health 0 [ 573.343400] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 573.346961] LNet: Skipped 3 previous similar messages [ 574.214912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 583.657454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 583.674591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 583.691906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 586.495940] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 14:16:40 (1758824200) [ 586.714245] LNet: 60956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.717069] LNetError: 60956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.720230] LNetError: 60956:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 586.726323] LNet: Removed LNI 192.168.203.51@tcp [ 586.736493] LNet: Removed LNI 192.168.203.51@tcp1 [ 587.014861] Key type .llcrypt unregistered [ 587.016043] Key type ._llcrypt unregistered [ 589.312471] Key type ._llcrypt registered [ 589.314255] Key type .llcrypt registered [ 589.394391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 589.401733] alg: No test for adler32 (adler32-zlib) [ 590.222224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 590.234669] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 593.884670] Lustre: DEBUG MARKER: Initial discovery [ 593.901684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 594.849377] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 594.870615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 594.889311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 594.900063] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 594.910837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 594.926806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.151@tcp -r 1 -m GET -l 3 [ 594.945578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.151@tcp -r 1 -m GET -e local_error [ 594.964089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.151@tcp -r 1 -m PUT -l 6 [ 595.504419] Lustre: DEBUG MARKER: Force oleg351-client.virtnet to discover 192.168.203.151@tcp (in background) [ 595.538236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.151@tcp [ 596.071175] Lustre: DEBUG MARKER: Force oleg351-server to discover 192.168.203.51@tcp [ 596.946382] Lustre: DEBUG MARKER: Wait for 61706 [ 598.559328] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 599.122883] Lustre: DEBUG MARKER: Finished wait on 61706 [ 599.173837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 599.194385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 599.304118] LNet: 62160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 599.307820] LNetError: 62160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 599.321723] LNet: Removed LNI 192.168.203.51@tcp [ 599.498627] Key type .llcrypt unregistered [ 599.500240] Key type ._llcrypt unregistered [ 605.351715] Key type ._llcrypt registered [ 605.353349] Key type .llcrypt registered [ 605.870942] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 14:16:59 (1758824219) [ 606.066317] Key type .llcrypt unregistered [ 606.067991] Key type ._llcrypt unregistered [ 608.198420] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 608.233246] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 608.235625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 608.239978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 610.463340] Key type ._llcrypt registered [ 610.464660] Key type .llcrypt registered [ 610.538715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 610.545567] alg: No test for adler32 (adler32-zlib) [ 611.359038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.385501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.398118] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 611.674815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 611.684387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.963148] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 14:17:08 (1758824228) [ 615.013582] LNet: 64218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.017873] LNetError: 64218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 615.021253] LNet: Removed LNI 192.168.203.51@tcp [ 615.028218] LNet: Removed LNI 10.1.2.3@tcp [ 615.285410] Key type .llcrypt unregistered [ 615.287060] Key type ._llcrypt unregistered [ 617.659134] Key type ._llcrypt registered [ 617.660722] Key type .llcrypt registered [ 617.747838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 617.754867] alg: No test for adler32 (adler32-zlib) [ 618.569854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.598551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 618.610683] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 618.903654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 618.915971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 619.324227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.51@tcp [ 620.383162] LNetError: Unexpected error -22 connecting to 192.168.203.51@tcp at host 192.168.203.51:988 [ 621.724756] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 14:17:15 (1758824235) [ 621.782849] LNet: 65339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.786202] LNetError: 65339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.789684] LNet: Removed LNI 192.168.203.51@tcp [ 621.796328] LNet: Removed LNI 10.1.2.3@tcp [ 622.057403] Key type .llcrypt unregistered [ 622.059038] Key type ._llcrypt unregistered [ 624.285590] Key type ._llcrypt registered [ 624.287186] Key type .llcrypt registered [ 624.367258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 624.375028] alg: No test for adler32 (adler32-zlib) [ 625.193642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 625.223364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 625.239912] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 625.537321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 625.547833] LNet: Added LNI 192.168.203.51@tcp2 [8/256/0/180] [ 625.906854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.51@tcp1 --nid 192.168.203.51@tcp2 [ 627.619467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp1 192.168.203.51@tcp2 [ 627.629922] LNetError: 66166:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.51@tcp1 to 192.168.203.51@tcp2 - Route table may be misconfigured [ 627.641482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp2 192.168.203.51@tcp1 [ 630.081874] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 14:17:23 (1758824243) [ 630.139272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.151211] LNet: 66738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.154714] LNetError: 66738:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.158943] LNet: Removed LNI 192.168.203.51@tcp1 [ 631.205626] LNet: Removed LNI 192.168.203.51@tcp2 [ 631.217409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.244948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 631.259697] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 631.556069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 631.566652] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 631.912379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 631.930723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.51@tcp [ 631.948364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp1 192.168.203.51@tcp1 [ 631.965840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.51@tcp -d 192.168.203.51@tcp -e network_timeout [ 631.983377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -e network_timeout [ 632.001604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -e network_timeout [ 632.021065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -e network_timeout [ 632.038900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 632.049738] LNet: There was an unexpected network error while writing to 192.168.203.51: rc = -22 [ 632.060584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 632.102871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 632.143101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 632.287111] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 634.545086] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 14:17:28 (1758824248) [ 634.594084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.605222] LNet: 67591:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.609907] LNetError: 67591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.615531] LNetError: 67591:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 634.618944] LNet: Removed LNI 192.168.203.51@tcp [ 635.685997] LNet: Removed LNI 192.168.203.51@tcp1 [ 635.698224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.745634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 635.860768] LNet: 67629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.040509] Key type .llcrypt unregistered [ 636.042070] Key type ._llcrypt unregistered [ 640.083558] Key type ._llcrypt registered [ 640.085300] Key type .llcrypt registered [ 640.615655] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 14:17:34 (1758824254) [ 640.760457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 640.767781] alg: No test for adler32 (adler32-zlib) [ 641.583121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.136741] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 642.766549] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 14:17:36 (1758824256) [ 642.796333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 642.809062] LNet: 68716:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.821114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 642.851190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 642.864264] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 643.162186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 643.172498] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 643.525767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 643.543906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp1 [ 643.562981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp1 [ 645.959133] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 14:17:39 (1758824259) [ 646.684321] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg351-server [ 647.354261] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 14:17:41 (1758824261) [ 648.078938] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg351-server [ 648.745990] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 14:17:42 (1758824262) [ 649.457202] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg351-server [ 650.135470] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 14:17:44 (1758824264) [ 650.869608] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg351-server [ 651.547734] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 14:17:45 (1758824265) [ 652.235955] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg351-server [ 652.876805] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 14:17:46 (1758824266) [ 653.574489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg351-server [ 654.224343] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 14:17:48 (1758824268) [ 654.958163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg351-server [ 655.630217] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 14:17:49 (1758824269) [ 656.324205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg351-server [ 656.957420] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 14:17:50 (1758824270) [ 657.625391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg351-server [ 658.244681] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 14:17:52 (1758824272) [ 658.283174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.294578] LNet: 72584:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.298450] LNetError: 72584:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.301930] LNet: Removed LNI 192.168.203.51@tcp [ 659.365407] LNet: Removed LNI 192.168.203.51@tcp1 [ 659.372613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.391385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.400136] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 659.634914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 659.666385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 659.729343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 659.737656] LNet: 72732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.740874] LNetError: 72732:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 659.743495] LNetError: 72732:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 659.756338] LNet: Removed LNI 192.168.203.51@tcp [ 659.765522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 659.788345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 659.799060] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 660.068666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 660.136514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.157438] LNet: Removed LNI 192.168.203.51@tcp [ 661.165512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.189131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 661.200421] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 661.457930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 661.490403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 661.560541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.570572] LNetError: 73019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.573779] LNetError: 73019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 661.583395] LNet: Removed LNI 192.168.203.51@tcp [ 661.591746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 661.613974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 661.624451] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 661.908662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 661.982530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.992475] LNet: 73161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.995361] LNet: 73161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 663.013336] LNet: Removed LNI 192.168.203.51@tcp [ 663.020789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.043213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.055204] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 663.306720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 663.339425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 663.422298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.485321] LNet: Removed LNI 192.168.203.51@tcp [ 664.494199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.518463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 664.529839] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 664.813019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 664.890625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.900462] LNetError: 73447:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.903739] LNetError: 73447:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 664.910494] LNet: Removed LNI 192.168.203.51@tcp [ 664.918440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.940753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.951622] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 665.234149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 665.266381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 665.350546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.405418] LNet: Removed LNI 192.168.203.51@tcp [ 666.413914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 666.439196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 666.451493] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 666.752896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 666.843966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 666.857317] LNet: 73734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 666.860558] LNet: 73734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 667.878445] LNet: Removed LNI 192.168.203.51@tcp [ 667.888521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 667.917341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 667.930507] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 668.240791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 668.279107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 669.470169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.481161] LNetError: 73888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.488636] LNetError: 73888:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 669.498744] LNet: Removed LNI 192.168.203.51@tcp [ 669.511109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.543840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 669.557768] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 669.873766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 669.999578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.077629] LNet: Removed LNI 192.168.203.51@tcp [ 671.087296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.116087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 671.132774] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 671.420481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 671.452880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 671.577866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.613379] LNet: Removed LNI 192.168.203.51@tcp [ 672.622833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.650622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 672.663763] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 672.964535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 675.188840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.199647] LNet: 74335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.201925] LNet: 74335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 676.261749] LNet: Removed LNI 192.168.203.51@tcp [ 676.270441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.294307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.305656] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 676.574924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 676.600747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.615312] LNet: Removed LNI 192.168.203.51@tcp [ 676.624104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.647458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.658306] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 676.936670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 679.496109] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 14:18:13 (1758824293) [ 679.543290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.554371] LNetError: 75163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.557737] LNetError: 75163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 679.566742] LNet: Removed LNI 192.168.203.51@tcp [ 679.575232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.601397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 679.613437] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 680.074319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.091467] LNet: Removed LNI 192.168.203.51@tcp231 [ 680.101477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.130732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 680.144608] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 680.146621] LNet: 75308:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 680.332687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 680.350030] LNet: Removed LNI 192.168.203.51@tcp231 [ 680.358920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 680.374773] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 680.851848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 680.871528] LNet: Removed LNI 192.168.203.51@tcp231 [ 680.882775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 680.913642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 680.925600] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/0] [ 680.929295] LNet: 75457:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 681.104562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 681.124467] LNet: Removed LNI 192.168.203.51@tcp231 [ 681.132198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 681.144826] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/0] [ 681.593756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 681.614452] LNet: Removed LNI 192.168.203.51@tcp231 [ 681.624826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.658121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 681.670987] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/60] [ 681.845150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 681.860663] LNet: Removed LNI 192.168.203.51@tcp231 [ 681.868575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 681.880289] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/60] [ 682.321983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.344377] LNet: Removed LNI 192.168.203.51@tcp231 [ 682.353874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.383732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 682.399333] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 682.402241] LNet: 75753:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 682.405456] LNet: 75753:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 682.575071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 682.595839] LNet: Removed LNI 192.168.203.51@tcp231 [ 682.603844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 682.616762] LNet: Added LNI 192.168.203.51@tcp231 [8/256/0/180] [ 685.463891] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 14:18:19 (1758824299) [ 685.543332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.558689] LNet: Removed LNI 192.168.203.51@tcp231 [ 685.568582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.599125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 685.613939] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 685.903142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 685.922395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 685.946637] LNet: Removed LNI 192.168.203.51@tcp232 [ 685.954657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 685.967169] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 686.265251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 686.283868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 686.303376] LNet: Removed LNI 192.168.203.51@tcp232 [ 686.312373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 686.325590] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 686.616659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 686.635862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 686.651755] LNet: Removed LNI 192.168.203.51@tcp232 [ 686.659681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 686.673596] LNet: Added LNI 192.168.203.51@tcp232 [8/256/0/180] [ 686.972944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 686.991696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 687.012357] LNet: Removed LNI 192.168.203.51@tcp232 [ 687.020391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 689.473108] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 14:18:23 (1758824303) [ 689.914916] Key type .llcrypt unregistered [ 689.916463] Key type ._llcrypt unregistered [ 692.219695] Key type ._llcrypt registered [ 692.221187] Key type .llcrypt registered [ 692.306318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 692.313889] alg: No test for adler32 (adler32-zlib) [ 693.123349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 693.136478] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 696.409338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 699.592453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.616377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 699.627710] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 700.201734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_interrupt -r 2 [ 700.221869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 2 [ 700.241517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_interrupt -r 2 [ 700.259534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 2 [ 700.278390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_interrupt -r 2 [ 700.295788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_interrupt -r 2 [ 700.314474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_interrupt -r 2 [ 700.332503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_interrupt -r 2 [ 700.351377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 700.362521] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 700.365635] LNet: 1 local NIs in recovery (showing 1): 192.168.203.51@tcp [ 700.495033] LNet: 78221:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 700.529675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 700.543305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 700.772842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_dropped -r 2 [ 700.790500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 2 [ 700.809329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_dropped -r 2 [ 700.828733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 2 [ 700.848342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_dropped -r 2 [ 700.867720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_dropped -r 2 [ 700.887894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_dropped -r 2 [ 700.907291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_dropped -r 2 [ 700.927545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 700.940890] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 701.079290] LNet: 78329:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 701.115300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 701.133135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 701.383199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_aborted -r 2 [ 701.400354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 2 [ 701.418500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_aborted -r 2 [ 701.437291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 2 [ 701.458056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_aborted -r 2 [ 701.476391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_aborted -r 2 [ 701.496394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_aborted -r 2 [ 701.515124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_aborted -r 2 [ 701.535679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 701.728948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 701.746515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 701.961716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_no_route -r 2 [ 701.979874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 2 [ 701.998444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_no_route -r 2 [ 702.014466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 2 [ 702.032369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_no_route -r 2 [ 702.050897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_no_route -r 2 [ 702.069262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_no_route -r 2 [ 702.088978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_no_route -r 2 [ 702.106818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 702.119710] LNet: There was an unexpected network error while writing to 192.168.203.151: rc = -22 [ 702.123052] LNet: Skipped 2 previous similar messages [ 702.257108] LNet: 78545:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 702.260376] LNet: 78545:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 702.288588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 702.304231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 702.523422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e local_timeout -r 2 [ 702.541397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 2 [ 702.557934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e local_timeout -r 2 [ 702.576150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 2 [ 702.594511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e local_timeout -r 2 [ 702.612489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e local_timeout -r 2 [ 702.633158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e local_timeout -r 2 [ 702.653326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e local_timeout -r 2 [ 702.672051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 702.860331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 702.875917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 703.102822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp -m GET -e remote_dropped -r 2 [ 703.124187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 2 [ 703.144995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp -m GET -e remote_dropped -r 2 [ 703.162953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.51@tcp -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 2 [ 703.180791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp -m GET -e remote_dropped -r 2 [ 703.198948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.151@tcp1 -m GET -e remote_dropped -r 2 [ 703.218935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp -m GET -e remote_dropped -r 2 [ 703.237917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.51@tcp1 -d 192.168.203.51@tcp1 -m GET -e remote_dropped -r 2 [ 703.254844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.51@tcp 192.168.203.151@tcp [ 703.266446] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.151@tcp [ 703.430151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 703.445646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 705.783093] LNet: 78830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.785962] LNetError: 78830:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.790502] LNet: Removed LNI 192.168.203.51@tcp [ 705.797375] LNet: Removed LNI 192.168.203.51@tcp1 [ 705.978673] Key type .llcrypt unregistered [ 705.979939] Key type ._llcrypt unregistered [ 711.882777] Key type ._llcrypt registered [ 711.884311] Key type .llcrypt registered [ 712.398581] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 14:18:46 (1758824326) [ 712.537310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 712.543549] alg: No test for adler32 (adler32-zlib) [ 713.356467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.380748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 713.393632] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 713.683369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 713.710294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 713.727579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 713.757415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 713.779080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 713.798446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 716.193472] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 14:18:50 (1758824330) [ 716.229334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 716.239146] LNet: 80379:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 716.242169] LNetError: 80379:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 716.254904] LNet: Removed LNI 192.168.203.51@tcp [ 716.266597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.291966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.303883] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 719.048443] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 14:18:52 (1758824332) [ 719.621272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 720.287299] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 14:18:54 (1758824334) [ 720.493556] LNet: 81414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.499182] LNetError: 81414:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.510194] LNet: Removed LNI 192.168.203.51@tcp [ 720.777681] Key type .llcrypt unregistered [ 720.779146] Key type ._llcrypt unregistered [ 723.000956] Key type ._llcrypt registered [ 723.002306] Key type .llcrypt registered [ 723.082296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 723.089307] alg: No test for adler32 (adler32-zlib) [ 723.900199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 723.912678] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 727.125460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 728.869382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.151@tcp [ 728.995541] LNet: 81824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.998540] LNetError: 81824:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 729.012636] LNet: Removed LNI 192.168.203.51@tcp [ 729.176653] Key type .llcrypt unregistered [ 729.178475] Key type ._llcrypt unregistered [ 733.661906] Key type ._llcrypt registered [ 733.663232] Key type .llcrypt registered [ 734.243363] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 734.901679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 735.653583] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 14:19:09 (1758824349) [ 735.810522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 735.817393] alg: No test for adler32 (adler32-zlib) [ 736.645397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.681721] LNet: 82906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.986439] Key type .llcrypt unregistered [ 736.988257] Key type ._llcrypt unregistered [ 739.327527] Key type ._llcrypt registered [ 739.329043] Key type .llcrypt registered [ 739.404736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.413133] alg: No test for adler32 (adler32-zlib) [ 740.240368] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 740.242917] LNet: 83158:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 740.487743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 758.943169] LNetError: 83170:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.55@tcp has gone from up to down [ 758.950192] LNet: 83216:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 758.953615] LNetError: 83216:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 758.963708] LNet: Removed LNI 192.168.203.51@tcp [ 758.983394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 759.016384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 759.031615] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 759.034646] LNet: 83225:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 759.038815] LNet: 83225:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 759.042621] LNetError: 83225:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.53@tcp is a gateway. Can not delete it [ 759.046312] LNetError: 83225:0:(peer.c:2201:lnet_del_peer_ni()) 192.168.203.53@tcp is a router. Can not be deleted [ 759.051945] LNet: 83225:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 762.141413] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 14:19:35 (1758824375) [ 762.143265] LNetError: 83226:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) route to o2ib through 192.168.203.52@tcp has gone from up to down [ 762.161048] LNetError: 83226:0:(lib-lnet.h:1315:lnet_set_route_aliveness()) Skipped 5 previous similar messages [ 763.023877] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg351-server [ 763.844728] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 14:19:37 (1758824377) [ 764.657042] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg351-server [ 765.369309] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 14:19:39 (1758824379) [ 766.042165] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 767.751152] Key type lgssc registered [ 768.256062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 777.163760] Key type lgssc unregistered [ 777.324189] LNet: 85641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 777.328373] LNetError: 85641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 777.341643] LNet: Removed LNI 192.168.203.51@tcp [ 777.586708] Key type .llcrypt unregistered [ 777.588470] Key type ._llcrypt unregistered [ 783.311303] Key type ._llcrypt registered [ 783.312761] Key type .llcrypt registered [ 783.889866] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 14:19:57 (1758824397) [ 784.051311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 784.059111] alg: No test for adler32 (adler32-zlib) [ 784.886878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.921859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 784.944338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 784.964878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 784.987611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 787.466545] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 14:20:01 (1758824401) [ 787.507522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.519199] LNet: 87035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.530640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.557044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 787.567409] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 787.920098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 787.929791] LNetError: 87181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.939447] LNet: Removed LNI 192.168.203.51@tcp [ 787.949533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 787.975490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 787.988511] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 788.180129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 788.190055] LNet: 87209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.193280] LNet: 87209:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 788.200534] LNet: Removed LNI 192.168.203.51@tcp [ 788.210190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 788.237171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 788.250579] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 790.714465] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 14:20:04 (1758824404) [ 790.758436] LNet: 87793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 790.762148] LNetError: 87793:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 790.765563] LNetError: 87793:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 790.775567] LNet: Removed LNI 192.168.203.51@tcp [ 791.054653] Key type .llcrypt unregistered [ 791.056408] Key type ._llcrypt unregistered [ 793.339082] Key type ._llcrypt registered [ 793.340776] Key type .llcrypt registered [ 793.419642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 793.427598] alg: No test for adler32 (adler32-zlib) [ 794.250356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 794.283851] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 794.285679] LNet: *** cfs_fail_loc=e003, val=100*** [ 794.597029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 794.608157] Ping reply received with mapping: [ 794.610149] found metadata reply NI[192.168.203.51@tcp] [ 794.612129] 33 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 794.614146] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 794.615930] 5a 5a 5a 5a 5a 5a 5a 5a [ 794.672844] LNet: 88188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 794.676246] LNetError: 88188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 794.685489] LNet: Removed LNI 192.168.203.51@tcp [ 794.957422] Key type .llcrypt unregistered [ 794.958826] Key type ._llcrypt unregistered [ 799.081475] Key type ._llcrypt registered [ 799.082986] Key type .llcrypt registered [ 799.611256] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 14:20:13 (1758824413) [ 799.835330] Key type .llcrypt unregistered [ 799.837034] Key type ._llcrypt unregistered [ 802.182118] Key type ._llcrypt registered [ 802.183704] Key type .llcrypt registered [ 802.265457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 802.274100] alg: No test for adler32 (adler32-zlib) [ 803.171551] LNetError: 89222:0:(api-ni.c:2710:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 803.341527] Key type .llcrypt unregistered [ 803.343136] Key type ._llcrypt unregistered [ 805.232673] Key type ._llcrypt registered [ 805.234202] Key type .llcrypt registered [ 805.750685] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 14:20:19 (1758824419) [ 805.967514] Key type .llcrypt unregistered [ 805.969034] Key type ._llcrypt unregistered [ 808.136342] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.138933] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 810.310232] Key type ._llcrypt registered [ 810.311770] Key type .llcrypt registered [ 810.380228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 810.386690] alg: No test for adler32 (adler32-zlib) [ 811.197900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 811.224553] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 815.443244] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 14:20:29 (1758824429) [ 815.497966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.509401] LNet: 90988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.512911] LNetError: 90988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 815.527152] LNet: Removed LNI 10.1.2.3@tcp [ 815.537163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.849674] Key type .llcrypt unregistered [ 815.851097] Key type ._llcrypt unregistered [ 817.967850] Key type ._llcrypt registered [ 817.969153] Key type .llcrypt registered [ 818.030179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.036414] alg: No test for adler32 (adler32-zlib) [ 818.856345] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 818.882607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 818.941229] LNet: 91273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.944127] LNetError: 91273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.952293] LNet: Removed LNI 192.168.203.51@tcp [ 819.205506] Key type .llcrypt unregistered [ 819.207115] Key type ._llcrypt unregistered [ 823.115073] Key type ._llcrypt registered [ 823.116448] Key type .llcrypt registered [ 823.626249] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 14:20:37 (1758824437) [ 823.865428] Key type .llcrypt unregistered [ 823.867061] Key type ._llcrypt unregistered [ 826.091368] Key type ._llcrypt registered [ 826.092923] Key type .llcrypt registered [ 826.185618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 826.193475] alg: No test for adler32 (adler32-zlib) [ 829.579912] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 14:20:43 (1758824443) [ 829.628791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 829.644867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 829.674311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 829.688107] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 829.983885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 830.000050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 830.015878] LNet: Removed LNI 192.168.203.51@tcp [ 830.117528] LNet: 93003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 830.311370] Key type .llcrypt unregistered [ 830.313049] Key type ._llcrypt unregistered [ 834.455451] Key type ._llcrypt registered [ 834.456608] Key type .llcrypt registered [ 835.012756] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 14:20:48 (1758824448) [ 835.156114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 835.165404] alg: No test for adler32 (adler32-zlib) [ 835.973968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.999119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.010043] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 836.286078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 836.301529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.51@tcp [ 836.310421] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 836.312995] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 836.314941] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 836, 0, 0 [ 836.317974] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 836.319707] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 836.321985] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 836.324104] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 836, 0, 0 [ 836.327125] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 836.328954] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 836.331196] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 836.333240] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 836, 0, 886 [ 836.336288] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 836.338034] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 836.340170] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 836.342187] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 836, 0, 886 [ 836.345142] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 838.708930] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 14:20:52 (1758824452) [ 838.875109] LNet: 94535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.879913] LNetError: 94535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 838.887537] LNet: Removed LNI 192.168.203.51@tcp [ 839.145505] Key type .llcrypt unregistered [ 839.147234] Key type ._llcrypt unregistered [ 841.262094] Key type ._llcrypt registered [ 841.263797] Key type .llcrypt registered [ 841.333084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 841.339645] alg: No test for adler32 (adler32-zlib) [ 842.154629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 842.170396] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 845.341244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 848.470454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.488773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 848.499506] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 848.892878] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 848.928214] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 848.930243] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 848.932337] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 851.102288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 851.113291] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 855.088968] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 14:21:08 (1758824468) [ 855.157156] LNet: 96029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.162267] LNetError: 96029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.166554] LNet: Removed LNI 192.168.203.51@tcp [ 856.223194] LNet: Removed LNI 192.168.203.51@tcp1 [ 856.224565] LNetError: 96029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 856.227664] LNetError: 96029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 856.236272] LNet: Removed LNI 10.1.2.3@tcp99 [ 856.510900] Key type .llcrypt unregistered [ 856.512309] Key type ._llcrypt unregistered [ 858.644137] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 858.646591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 860.805948] Key type ._llcrypt registered [ 860.807477] Key type .llcrypt registered [ 860.876960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 860.883218] alg: No test for adler32 (adler32-zlib) [ 861.692721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.716988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 861.728178] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 862.000540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 862.010346] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.824941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 862.847859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 862.866395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 862.876600] LNetError: 96763:0:(peer.c:2209:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 862.890950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 862.908692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 863.065385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 863.095894] LNet: 96783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.099323] LNetError: 96783:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 863.102424] LNet: Removed LNI 192.168.203.51@tcp [ 863.112407] LNet: Removed LNI 10.1.2.3@tcp [ 863.375529] Key type .llcrypt unregistered [ 863.377096] Key type ._llcrypt unregistered [ 867.276622] Key type ._llcrypt registered [ 867.278393] Key type .llcrypt registered [ 867.773670] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 14:21:21 (1758824481) [ 867.911371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 867.916657] alg: No test for adler32 (adler32-zlib) [ 868.721383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 868.743347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 868.754727] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 869.013953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.51@tcp [ 869.044471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg351-client@tcp [ 871.267974] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 14:21:25 (1758824485) [ 871.303879] LNet: 98299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.308175] LNetError: 98299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.315607] LNet: Removed LNI 192.168.203.51@tcp [ 871.578414] Key type .llcrypt unregistered [ 871.579610] Key type ._llcrypt unregistered [ 873.574296] Key type ._llcrypt registered [ 873.575545] Key type .llcrypt registered [ 873.635496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 873.642368] alg: No test for adler32 (adler32-zlib) [ 874.444812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 874.455370] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 874.476498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 876.724664] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 14:21:30 (1758824490) [ 877.272081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 877.876650] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 14:21:31 (1758824491) [ 878.417123] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 879.006433] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 14:21:32 (1758824492) [ 879.047497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 879.058338] LNet: 99715:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.060890] LNetError: 99715:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.073402] LNet: Removed LNI 192.168.203.51@tcp [ 879.083337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 879.106498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 879.117780] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 879.373126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 879.392228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 881.699613] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 14:21:35 (1758824495) [ 881.739432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 881.750761] LNet: 100420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.753750] LNetError: 100420:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.764966] LNet: Removed LNI 192.168.203.51@tcp [ 881.776745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.800887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 881.818230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 882.070450] Key type .llcrypt unregistered [ 882.072037] Key type ._llcrypt unregistered [ 885.890347] Key type ._llcrypt registered [ 885.891774] Key type .llcrypt registered [ 886.407365] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 14:21:40 (1758824500) [ 886.537847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.543974] alg: No test for adler32 (adler32-zlib) [ 887.359926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.382825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 887.393449] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 887.666850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 887.691178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 890.039388] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 14:21:43 (1758824503) [ 890.076469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.086638] LNet: 101966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.089869] LNetError: 101966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.099521] LNet: Removed LNI 192.168.203.51@tcp [ 890.113383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 890.135324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 890.148369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 892.388137] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 893.008267] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 14:21:46 (1758824506) [ 893.048032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.059294] LNet: 102686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.070503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.185104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 893.195475] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 893.480210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 893.491028] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 893.822649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.51@tcp [ 893.839463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 893.866375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.51@tcp -m PUT -e local_timeout -r 1 [ 893.892979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 893.919716] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 893.921615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 894.019841] LNetError: 102722:0:(lib-move.c:4947:lnet_parse()) 192.168.203.51@tcp, src 192.168.203.51@tcp: Dropping PUT (error -108 looking up sender) [ 895.071440] LNet: 102998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.074735] LNetError: 102998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 895.077922] LNet: Removed LNI 192.168.203.51@tcp [ 895.087305] LNet: Removed LNI 10.1.2.3@tcp [ 895.260540] Key type .llcrypt unregistered [ 895.261752] Key type ._llcrypt unregistered [ 899.171292] Key type ._llcrypt registered [ 899.172806] Key type .llcrypt registered [ 899.679947] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 14:21:53 (1758824513) [ 899.807237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 899.813612] alg: No test for adler32 (adler32-zlib) [ 900.626235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 900.785242] LNet: 103847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.040542] Key type .llcrypt unregistered [ 901.042062] Key type ._llcrypt unregistered [ 903.327852] Key type ._llcrypt registered [ 903.329378] Key type .llcrypt registered [ 903.403335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.409919] alg: No test for adler32 (adler32-zlib) [ 904.220546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 904.234054] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 907.274345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 907.336861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 907.563799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.151@tcp [ 907.705776] LNet: 104285:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 907.740692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 907.756025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.002648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 908.153564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.168991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.393057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.151@tcp [ 908.507898] LNet: 104495:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 908.510950] LNet: 104495:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 908.537287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.550982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 908.766645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 908.900831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 908.913890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 909.019619] LNet: 104641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.022505] LNetError: 104641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.035347] LNet: Removed LNI 192.168.203.51@tcp [ 909.180638] Key type .llcrypt unregistered [ 909.181929] Key type ._llcrypt unregistered [ 914.700554] Key type ._llcrypt registered [ 914.701970] Key type .llcrypt registered [ 915.237394] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 14:22:09 (1758824529) [ 915.388399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 915.396479] alg: No test for adler32 (adler32-zlib) [ 916.214991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.363574] LNet: 105523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.613420] Key type .llcrypt unregistered [ 916.615111] Key type ._llcrypt unregistered [ 918.658305] Key type ._llcrypt registered [ 918.659705] Key type .llcrypt registered [ 918.733708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 918.740038] alg: No test for adler32 (adler32-zlib) [ 919.558554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 919.571643] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 922.518901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.151@tcp [ 925.295721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.310161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 925.320738] LNet: Added LNI 192.168.203.51@tcp1 [8/256/0/180] [ 925.667521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 925.902319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.151@tcp [ 926.028038] LNet: 106157:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.059974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.076151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.338705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 926.515101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.532746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 926.792405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.151@tcp [ 926.931708] LNet: 106367:0:(api-ni.c:339:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 926.934970] LNet: 106367:0:(api-ni.c:339:recovery_interval_set()) Skipped 1 previous similar message [ 926.963671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 926.978903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 927.221524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 927.379577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 927.393683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.573502] LNet: 106548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.577051] LNetError: 106548:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.581167] LNet: Removed LNI 192.168.203.51@tcp [ 929.589539] LNet: Removed LNI 192.168.203.51@tcp1 [ 929.754627] Key type .llcrypt unregistered [ 929.755784] Key type ._llcrypt unregistered [ 935.140760] Key type ._llcrypt registered [ 935.142427] Key type .llcrypt registered [ 935.623719] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 819 sec =========== 14:22:29 (1758824549) [ 935.852550] Key type .llcrypt unregistered [ 935.853787] Key type ._llcrypt unregistered [ 939.871379] Key type ._llcrypt registered [ 939.873153] Key type .llcrypt registered [ 940.061711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.070996] alg: No test for adler32 (adler32-zlib) [ 940.953469] Lustre: Lustre: Build Version: 2.16.58_75_gd19a518 [ 941.058404] LNet: Added LNI 192.168.203.51@tcp [8/256/0/180] [ 942.647172] Key type lgssc registered [ 943.159249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 966.957370] Lustre: Mounted lustre-client [ 969.518557] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 977.023311] LustreError: 109757:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff88c646348800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 977.032448] LustreError: 109757:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 977.045924] Lustre: Unmounted lustre-client [ 989.534270] Key type lgssc unregistered [ 989.662038] LNet: 110238:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.665484] LNetError: 110238:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.677308] LNet: Removed LNI 192.168.203.51@tcp [ 989.965113] Key type .llcrypt unregistered [ 989.966173] Key type ._llcrypt unregistered