[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 459048839 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001018] APIC: Switch to symmetric I/O mode setup [ 0.002383] x2apic enabled [ 0.003015] Switched APIC routing to physical x2apic. [ 0.004017] kvm-guest: setup PV IPIs [ 0.007730] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009019] pid_max: default: 32768 minimum: 301 [ 0.011073] LSM: Security Framework initializing [ 0.013068] Yama: becoming mindful. [ 0.014051] SELinux: Initializing. [ 0.015100] *** VALIDATE selinux *** [ 0.023402] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028401] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030156] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031141] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033161] *** VALIDATE tmpfs *** [ 0.035100] *** VALIDATE proc *** [ 0.036300] *** VALIDATE cgroup *** [ 0.037013] *** VALIDATE cgroup2 *** [ 0.039319] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042043] Spectre V2 : User space: Vulnerable [ 0.043015] Speculative Store Bypass: Vulnerable [ 0.046818] debug: unmapping init [mem 0xffffffff9d859000-0xffffffff9d860fff] [ 0.048202] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049798] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050034] ... version: 2 [ 0.051024] ... bit width: 48 [ 0.052019] ... generic registers: 4 [ 0.053020] ... value mask: 0000ffffffffffff [ 0.054024] ... max period: 00007fffffffffff [ 0.055025] ... fixed-purpose events: 3 [ 0.056019] ... event mask: 000000070000000f [ 0.058306] rcu: Hierarchical SRCU implementation. [ 0.060657] smp: Bringing up secondary CPUs ... [ 0.061727] x86: Booting SMP configuration: [ 0.062047] .... node #0, CPUs: #1 #2 #3 [ 0.065785] smp: Brought up 1 node, 4 CPUs [ 0.067018] smpboot: Max logical packages: 1 [ 0.068022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.198419] node 0 deferred pages initialised in 128ms [ 0.201224] devtmpfs: initialized [ 0.202271] x86/mm: Memory block size: 128MB [ 0.204754] gcov: version magic: 0x41383552 [ 0.206373] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.210162] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.213453] pinctrl core: initialized pinctrl subsystem [ 0.214204] [ 0.214756] ************************************************************* [ 0.217024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.220019] ** ** [ 0.222016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.224021] ** ** [ 0.227019] ** This means that this kernel is built to expose internal ** [ 0.229015] ** IOMMU data structures, which may compromise security on ** [ 0.231018] ** your system. ** [ 0.233014] ** ** [ 0.236020] ** If you see this message and you are not debugging the ** [ 0.238017] ** kernel, report this immediately to your vendor! ** [ 0.240019] ** ** [ 0.242019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.244019] ************************************************************* [ 0.246880] NET: Registered protocol family 16 [ 0.248536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.251088] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.254078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.258079] cpuidle: using governor menu [ 0.259816] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.262615] PCI: Using configuration type 1 for base access [ 0.265147] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.274146] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.275029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.279078] cryptd: max_cpu_qlen set to 1000 [ 0.281334] ACPI: Added _OSI(Module Device) [ 0.283045] ACPI: Added _OSI(Processor Device) [ 0.285025] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.289023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.293803] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.301122] ACPI: Interpreter enabled [ 0.302088] ACPI: PM: (supports S0 S3 S4 S5) [ 0.304024] ACPI: Using IOAPIC for interrupt routing [ 0.305135] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.308450] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.318470] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.321067] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.323032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.326103] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.331365] acpiphp: Slot [2] registered [ 0.333186] acpiphp: Slot [3] registered [ 0.334109] acpiphp: Slot [4] registered [ 0.337157] acpiphp: Slot [5] registered [ 0.338204] acpiphp: Slot [6] registered [ 0.340158] acpiphp: Slot [7] registered [ 0.342124] acpiphp: Slot [8] registered [ 0.343106] acpiphp: Slot [9] registered [ 0.345098] acpiphp: Slot [10] registered [ 0.346126] acpiphp: Slot [11] registered [ 0.347119] acpiphp: Slot [12] registered [ 0.349113] acpiphp: Slot [13] registered [ 0.351113] acpiphp: Slot [14] registered [ 0.352111] acpiphp: Slot [15] registered [ 0.354119] acpiphp: Slot [16] registered [ 0.355106] acpiphp: Slot [17] registered [ 0.357156] acpiphp: Slot [18] registered [ 0.358115] acpiphp: Slot [19] registered [ 0.360114] acpiphp: Slot [20] registered [ 0.362131] acpiphp: Slot [21] registered [ 0.363117] acpiphp: Slot [22] registered [ 0.365130] acpiphp: Slot [23] registered [ 0.367130] acpiphp: Slot [24] registered [ 0.368126] acpiphp: Slot [25] registered [ 0.369189] acpiphp: Slot [26] registered [ 0.371128] acpiphp: Slot [27] registered [ 0.372117] acpiphp: Slot [28] registered [ 0.374135] acpiphp: Slot [29] registered [ 0.375131] acpiphp: Slot [30] registered [ 0.377138] acpiphp: Slot [31] registered [ 0.378082] PCI host bridge to bus 0000:00 [ 0.380030] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.382031] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.385034] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.387030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.389033] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.392038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.394212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399202] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.404489] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413559] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.417380] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.420026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.422021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.425034] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.427502] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.431122] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.433062] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.436731] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.441015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.450824] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.455026] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.460560] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.466031] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.472031] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.483023] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.491530] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.497019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.502023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.521032] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.534000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.537480] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.539450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.542426] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.544301] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.548136] iommu: Default domain type: Passthrough [ 0.551215] SCSI subsystem initialized [ 0.552158] ACPI: bus type USB registered [ 0.554140] usbcore: registered new interface driver usbfs [ 0.556123] usbcore: registered new interface driver hub [ 0.558115] usbcore: registered new device driver usb [ 0.560224] pps_core: LinuxPPS API ver. 1 registered [ 0.561010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.564073] PTP clock support registered [ 0.566201] EDAC MC: Ver: 3.0.0 [ 0.568138] PCI: Using ACPI for IRQ routing [ 0.569837] NetLabel: Initializing [ 0.571016] NetLabel: domain hash size = 128 [ 0.573013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.575121] NetLabel: unlabeled traffic allowed by default [ 0.577190] vgaarb: loaded [ 0.580025] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.581014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.585101] clocksource: Switched to clocksource kvm-clock [ 0.690822] VFS: Disk quotas dquot_6.6.0 [ 0.692331] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.695178] *** VALIDATE ramfs *** [ 0.696592] *** VALIDATE hugetlbfs *** [ 0.698397] pnp: PnP ACPI init [ 0.700967] pnp: PnP ACPI: found 6 devices [ 0.720839] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.723965] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.725839] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.727760] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.730326] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.732867] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.735565] NET: Registered protocol family 2 [ 0.738290] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.743552] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.749810] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.755493] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.759360] TCP: Hash tables configured (established 65536 bind 65536) [ 0.762480] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.764870] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.767862] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.770831] NET: Registered protocol family 1 [ 0.775242] RPC: Registered named UNIX socket transport module. [ 0.777517] RPC: Registered udp transport module. [ 0.779385] RPC: Registered tcp transport module. [ 0.781203] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.783594] NET: Registered protocol family 44 [ 0.785374] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.787832] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.789722] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.791904] PCI: CLS 0 bytes, default 64 [ 0.793611] Unpacking initramfs... [ 2.217763] debug: unmapping init [mem 0xffff991fbcc64000-0xffff991fbffcffff] [ 2.222046] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.224583] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.227830] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.742167] Initialise system trusted keyrings [ 2.743658] Key type blacklist registered [ 2.745391] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.756961] zbud: loaded [ 2.760284] *** VALIDATE nfs *** [ 2.761723] *** VALIDATE nfs4 *** [ 2.763658] pstore: using deflate compression [ 2.767519] Platform Keyring initialized [ 2.890359] NET: Registered protocol family 38 [ 2.892281] Key type asymmetric registered [ 2.894515] Asymmetric key parser 'x509' registered [ 2.896818] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.901264] io scheduler mq-deadline registered [ 2.903474] io scheduler kyber registered [ 2.905421] io scheduler bfq registered [ 2.907537] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.910757] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.913494] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.916482] ACPI: Power Button [PWRF] [ 3.009911] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.102699] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.197322] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.226447] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.258976] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.264155] Non-volatile memory driver v1.3 [ 3.266139] Linux agpgart interface v0.103 [ 3.298422] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.301658] vda: detected capacity change from 0 to 68530176 [ 3.321828] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.325821] vdb: detected capacity change from 0 to 1073741824 [ 3.334138] libphy: Fixed MDIO Bus: probed [ 3.340913] usbcore: registered new interface driver usbserial_generic [ 3.343100] usbserial: USB Serial support registered for generic [ 3.345043] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.348770] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.350417] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.352876] mousedev: PS/2 mouse device common for all mice [ 3.358804] rtc_cmos 00:05: RTC can wake from S4 [ 3.363191] rtc_cmos 00:05: registered as rtc0 [ 3.363225] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.365199] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.372790] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.373056] intel_pstate: CPU model not supported [ 3.376590] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.382465] hid: raw HID events driver (C) Jiri Kosina [ 3.384463] usbcore: registered new interface driver usbhid [ 3.386618] usbhid: USB HID core driver [ 3.388416] drop_monitor: Initializing network drop monitor service [ 3.390770] Initializing XFRM netlink socket [ 3.393026] NET: Registered protocol family 10 [ 3.397115] Segment Routing with IPv6 [ 3.398711] NET: Registered protocol family 17 [ 3.403575] mpls_gso: MPLS GSO support [ 3.411799] RAS: Correctable Errors collector initialized. [ 3.414078] AVX version of gcm_enc/dec engaged. [ 3.416091] AES CTR mode by8 optimization enabled [ 3.509330] sched_clock: Marking stable (3509301573, 0)->(4480643720, -971342147) [ 3.517118] registered taskstats version 1 [ 3.519121] Loading compiled-in X.509 certificates [ 3.521339] zswap: loaded using pool lzo/zbud [ 3.549698] Key type big_key registered [ 3.562090] Key type encrypted registered [ 3.563698] ima: No TPM chip found, activating TPM-bypass! [ 3.566237] ima: Allocated hash algorithm: sha1 [ 3.567496] ima: No architecture policies found [ 3.569206] evm: Initialising EVM extended attributes: [ 3.571613] evm: security.selinux [ 3.572783] evm: security.ima [ 3.574064] evm: security.capability [ 3.575441] evm: HMAC attrs: 0x1 [ 3.577593] rtc_cmos 00:05: setting system clock to 2025-11-16 23:49:21 UTC (1763336961) [ 3.583791] debug: unmapping init [mem 0xffffffff9e803000-0xffffffff9e9fffff] [ 3.586597] debug: unmapping init [mem 0xffffffff9d582000-0xffffffff9d858fff] [ 3.594110] Write protecting the kernel read-only data: 28672k [ 3.596937] debug: unmapping init [mem 0xffffffff9bc03000-0xffffffff9bdfffff] [ 3.599979] debug: unmapping init [mem 0xffffffff9c514000-0xffffffff9c5fffff] [ 3.633444] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.642519] systemd[1]: Detected virtualization kvm. [ 3.644617] systemd[1]: Detected architecture x86-64. [ 3.646665] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.674435] systemd[1]: No hostname configured. [ 3.676417] systemd[1]: Set hostname to . [ 3.678741] random: systemd: uninitialized urandom read (16 bytes read) [ 3.681484] systemd[1]: Initializing machine ID from random generator. [ 3.728803] random: ln: uninitialized urandom read (6 bytes read) [ 3.819312] random: systemd: uninitialized urandom read (16 bytes read) [ 3.822630] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.828779] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.833285] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.477286] device-mapper: uevent: version 1.0.3 [ 4.479580] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.217291] virtio_net virtio0 ens2: renamed from eth0 [ 5.274051] scsi host0: ata_piix [ 5.300445] scsi host1: ata_piix [ 5.302638] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.304767] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.013595] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.084605] random: crng init done [ 10.086275] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.597292] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.819102] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.090624] SELinux: Disabled at runtime. [ 12.152652] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.160816] systemd[1]: Detected virtualization kvm. [ 12.163439] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.736449] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.739651] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.745959] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.750394] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.754164] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.763058] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.769878] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 12.978782] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.291796] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.580121] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.642577] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.710604] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.732636] EDAC sbridge: Ver: 1.1.2 [ 15.081722] Key type dns_resolver registered [ 15.394819] NFS: Registering the id_resolver key type [ 15.396484] Key type id_resolver registered [ 15.397656] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg414-client login: [ 24.533057] hrtimer: interrupt took 2257128 ns [ 70.811224] libcfs: loading out-of-tree module taints kernel. [ 70.873784] Key type ._llcrypt registered [ 70.875017] Key type .llcrypt registered [ 71.592922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 71.608757] alg: No test for adler32 (adler32-zlib) [ 73.144943] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 74.023739] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 75.831209] Key type lgssc registered [ 77.485525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 185.709777] Lustre: Mounted lustre-client [ 190.289397] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.426417] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.427809] Lustre: lustre-OST0000-osc-ffff992009a9d800: disconnect after 24s idle [ 213.503503] Lustre: DEBUG MARKER: oleg414-client.virtnet: executing yml_node [ 218.306046] Lustre: DEBUG MARKER: Client: 2.16.61.50 [ 220.346421] Lustre: DEBUG MARKER: MDS: 2.16.61.50 [ 222.861713] Lustre: DEBUG MARKER: OSS: 2.16.61.50 [ 224.257842] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 18:53:00 EST 2025 [ 237.956938] LustreError: 7930:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff992009a9d800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 237.970214] LustreError: 7930:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 237.991982] Lustre: Unmounted lustre-client [ 264.476430] Key type lgssc unregistered [ 264.769944] LNet: 8377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 264.778735] LNetError: 8377:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 264.803394] LNet: Removed LNI 192.168.204.14@tcp [ 265.525504] Key type .llcrypt unregistered [ 265.527134] Key type ._llcrypt unregistered [ 286.418693] Key type ._llcrypt registered [ 286.422439] Key type .llcrypt registered [ 286.610973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 286.631865] alg: No test for adler32 (adler32-zlib) [ 287.565815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 287.639813] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 287.670485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 291.462398] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 291.565571] LNet: 9242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.569776] LNetError: 9242:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.596257] LNet: Removed LNI 192.168.204.14@tcp [ 292.426526] Key type .llcrypt unregistered [ 292.428432] Key type ._llcrypt unregistered [ 297.548687] Key type ._llcrypt registered [ 297.552522] Key type .llcrypt registered [ 298.632150] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:54:15 (1763337255) [ 298.953236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 298.971578] alg: No test for adler32 (adler32-zlib) [ 299.863527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.944268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.975351] LNet: 9656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.015193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.055576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.082940] LNet: 9663:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 305.952477] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:54:22 (1763337262) [ 306.023483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.037268] LNet: 10233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.081378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.163821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 311.522794] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:54:28 (1763337268) [ 311.599299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.619204] LNet: 10819:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.649918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.712101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 316.727708] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:54:33 (1763337273) [ 316.848228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.881921] LNet: 11407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.916397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.989638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 323.397237] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:54:39 (1763337279) [ 323.487942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.516610] LNet: 11988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.543643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.656347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 324.068476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 330.622556] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:54:47 (1763337287) [ 330.691956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.749116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.856884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 331.272860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 331.593273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 337.124485] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:54:53 (1763337293) [ 337.219047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.259862] LNet: 13179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.263690] LNet: 13179:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 337.299983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.413887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 343.886258] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:55:00 (1763337300) [ 344.113481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.152781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.236639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 344.264504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 344.748636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 344.788764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 345.163853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 345.199602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 345.609168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 345.661235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 346.105896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 346.138784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 346.572268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 346.631370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 346.994969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 347.046842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 347.501628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 347.530674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 347.950994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 347.991109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 348.387535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 348.447094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 348.810267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 348.858556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 349.233732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 349.273208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 349.691642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 349.720811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 354.927540] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:55:11 (1763337311) [ 355.033573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.055335] LNet: 14448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.061701] LNet: 14448:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 355.114509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.198585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 355.234695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 360.459921] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:55:17 (1763337317) [ 360.556321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.635514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.772511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 360.868666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 368.236570] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:55:24 (1763337324) [ 368.363155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.424747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.503808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 368.534220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 375.256061] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:55:31 (1763337331) [ 375.777592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.862608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.024763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 376.074107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 382.937696] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:55:39 (1763337339) [ 383.129288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.204424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.310475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 383.346407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 390.097684] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:55:46 (1763337346) [ 390.234536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.276641] LNet: 17377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 390.286270] LNet: 17377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 390.369581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.497441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 390.536972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 397.846551] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:55:54 (1763337354) [ 397.916688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.981370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.097991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 398.736233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 405.138515] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:56:01 (1763337361) [ 405.309323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.348595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.507509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 405.682779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 411.829191] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:56:08 (1763337368) [ 412.026739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.094836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.202631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.508861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.332442] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:56:15 (1763337375) [ 419.429108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.503901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.039888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.338588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 426.022635] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:56:22 (1763337382) [ 426.138075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.185479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.284150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 426.354570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 433.444982] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:56:29 (1763337389) [ 433.531484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.593934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.705496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.749883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 440.197300] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:56:36 (1763337396) [ 440.483910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.536622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.601530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 447.095260] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:56:43 (1763337403) [ 447.294459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.351941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.419957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 453.044825] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:56:49 (1763337409) [ 453.213606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.260224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.325152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 453.631980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 460.031333] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:56:56 (1763337416) [ 460.123095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.217247] LNet: 23403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.227926] LNet: 23403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 460.244145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.371634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 460.407470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 465.891856] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:57:02 (1763337422) [ 465.970451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.052516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.175349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 466.206399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 471.842916] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:57:08 (1763337428) [ 472.199192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 472.239766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 472.324361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 472.379551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 479.345791] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:57:15 (1763337435) [ 479.400441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.469934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.572077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 479.615466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 486.990548] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:57:23 (1763337443) [ 487.206851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.238160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.300669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 487.457066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 487.482969] LNetError: 25767:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 487.518170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 487.547053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 492.884173] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:57:29 (1763337449) [ 493.013277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 493.098471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 493.197186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mJmZxEfFP6 [ 493.272580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mJmZxEfFP6 [ 493.325495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.mJmZxEfFP6 [ 498.802758] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:57:35 (1763337455) [ 499.038807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.078381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.132801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 499.158199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 499.200925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 504.851093] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:57:41 (1763337461) [ 504.981472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.026703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.113868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 507.839874] LNet: Message forwarding will be enabled in 2s [ 510.945101] LNet: Message forwarding enabled [ 511.349487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 511.400351] LNet: Message forwarding will stop in 2s to 6s [ 515.887727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 516.264048] LNet: Message forwarding disabled [ 517.799189] LNet: Message forwarding will be enabled in 2s [ 521.376162] LNet: Message forwarding enabled [ 523.508199] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:58:00 (1763337480) [ 523.744762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.690990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.756696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 529.772203] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:58:06 (1763337486) [ 529.855540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 529.930420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.013059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 530.070350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 530.101772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 530.132707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 530.175647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 530.208575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 530.237025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 530.267059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 530.324283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 530.360264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 530.398695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 530.431537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 530.472117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 530.501332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 530.550352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 530.576642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 530.605372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 530.639611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 530.661808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 530.682047] LNetError: 28837:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 536.248412] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:58:12 (1763337492) [ 536.406292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.455874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.558406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.336952] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:58:19 (1763337499) [ 543.888239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg414-server [ 545.374800] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:58:22 (1763337502) [ 545.523603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 545.555130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.604520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 545.629337] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 546.932484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[15-19]@tcp [ 546.949415] LNet: 30552:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 547.006252] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 547.020397] LNetError: 30342:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.16@tcp has gone from up to down [ 548.003065] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.19@tcp is being used as a gateway but routing feature is not turned on [ 548.012516] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 549.047591] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 549.053716] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 551.085703] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.18@tcp is being used as a gateway but routing feature is not turned on [ 551.089572] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 552.925871] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:58:29 (1763337509) [ 554.881941] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg414-server [ 555.182930] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.15@tcp is being used as a gateway but routing feature is not turned on [ 555.204852] LNetError: 30342:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 556.563495] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:58:33 (1763337513) [ 556.772260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.820545] LNetError: 31487:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 556.843548] LNet: Removed LNI 192.168.204.14@tcp [ 556.858800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.906755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 556.940058] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 557.539714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[15-20/2]@tcp [ 557.577916] LNet: 31604:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 557.592589] LNet: 31604:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 557.616130] LNetError: 31491:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.15@tcp has gone from up to down [ 557.635691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[15-20/2]@tcp [ 557.644607] LNetError: 31491:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 563.747914] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:58:40 (1763337520) [ 563.852577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.871449] LNetError: 32190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.902407] LNet: Removed LNI 192.168.204.14@tcp [ 564.944725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.116350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 565.194778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.239395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.318571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.371889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.413911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.459253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 565.543621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.576692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.653398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 565.744654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.786229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.831455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 565.873130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.903729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 565.953706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 566.033428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.072312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.114608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 566.186239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.226736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.280605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 566.365725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.425992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.490548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 566.606967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.647223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.762087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 566.843313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.892951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.967055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 567.026518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.063767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.124959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 572.253430] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:58:49 (1763337529) [ 573.756848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg414-server [ 575.086995] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:58:51 (1763337531) [ 576.872310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg414-server [ 578.381934] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:58:55 (1763337535) [ 579.367351] Key type .llcrypt unregistered [ 579.370876] Key type ._llcrypt unregistered [ 585.325439] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 585.444085] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 585.446068] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 585.464103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 588.106363] Key type ._llcrypt registered [ 588.108868] Key type .llcrypt registered [ 588.279362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 588.290083] alg: No test for adler32 (adler32-zlib) [ 589.272245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 589.328469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 589.354453] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 590.041601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 590.070893] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 592.197561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 592.226253] LNet: Removed LNI 10.1.2.3@tcp [ 592.366614] LNet: 34315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 592.371453] LNetError: 34315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 592.385355] LNet: Removed LNI 192.168.204.14@tcp [ 592.932387] Key type .llcrypt unregistered [ 592.934220] Key type ._llcrypt unregistered [ 602.217984] Key type ._llcrypt registered [ 602.219692] Key type .llcrypt registered [ 603.319909] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:59:20 (1763337560) [ 603.761797] Key type .llcrypt unregistered [ 603.763664] Key type ._llcrypt unregistered [ 607.801919] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 607.941721] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 607.946089] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 607.952321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 610.481751] Key type ._llcrypt registered [ 610.483390] Key type .llcrypt registered [ 610.690610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 610.702510] alg: No test for adler32 (adler32-zlib) [ 611.581076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.626734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.648339] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 612.263541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 612.277199] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.187916] LNet: 35795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.197614] LNetError: 35795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.204973] LNet: Removed LNI 192.168.204.14@tcp [ 614.216825] LNet: Removed LNI 10.1.2.3@tcp [ 614.725588] Key type .llcrypt unregistered [ 614.727071] Key type ._llcrypt unregistered [ 624.459123] Key type ._llcrypt registered [ 624.460776] Key type .llcrypt registered [ 625.560715] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:59:42 (1763337582) [ 626.179593] Key type .llcrypt unregistered [ 626.181029] Key type ._llcrypt unregistered [ 630.892884] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 631.015602] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 631.018179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 631.020725] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 633.817494] Key type ._llcrypt registered [ 633.820650] Key type .llcrypt registered [ 634.008939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.028964] alg: No test for adler32 (adler32-zlib) [ 634.964047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.029443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 635.047520] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 636.835676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 636.860337] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 638.797140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 638.815952] LNet: Removed LNI 10.1.2.3@tcp [ 638.830403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 638.867673] LNet: Removed LNI 10.1.2.31@tcp [ 638.894771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 638.929822] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 639.721625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 639.746526] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 640.657832] LNet: 37709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.663555] LNetError: 37709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 640.668827] LNet: Removed LNI 10.1.2.3@tcp [ 640.682459] LNet: Removed LNI 10.1.2.31@tcp [ 641.257993] Key type .llcrypt unregistered [ 641.267072] Key type ._llcrypt unregistered [ 651.942182] Key type ._llcrypt registered [ 651.945697] Key type .llcrypt registered [ 653.239135] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 19:00:09 (1763337609) [ 653.517828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 653.532993] alg: No test for adler32 (adler32-zlib) [ 654.420086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.471730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 654.491527] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 655.141032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.157644] LNet: 38658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.170086] LNetError: 38658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.184542] LNet: Removed LNI 192.168.204.14@tcp [ 655.206671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.283768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.14@tcp [ 655.312676] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 656.210848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.14@tcp [ 656.245419] LNet: Removed LNI 192.168.204.14@tcp [ 656.298111] LNet: 38794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.895762] Key type .llcrypt unregistered [ 656.897490] Key type ._llcrypt unregistered [ 666.558721] Key type ._llcrypt registered [ 666.560311] Key type .llcrypt registered [ 667.797696] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 19:00:24 (1763337624) [ 669.282631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg414-server [ 670.755230] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 19:00:27 (1763337627) [ 671.212507] Key type .llcrypt unregistered [ 671.214648] Key type ._llcrypt unregistered [ 677.618424] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 677.840804] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 677.845034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 677.865332] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.374190] Key type ._llcrypt registered [ 680.375609] Key type .llcrypt registered [ 680.573892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 680.584365] alg: No test for adler32 (adler32-zlib) [ 681.527536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.630346] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 682.458890] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 684.841875] LNet: 40634:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.849962] LNetError: 40634:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.855182] LNet: Removed LNI 192.168.204.14@tcp1 [ 684.870888] LNet: Removed LNI 10.1.2.3@tcp2 [ 685.552618] Key type .llcrypt unregistered [ 685.555868] Key type ._llcrypt unregistered [ 695.373667] Key type ._llcrypt registered [ 695.380157] Key type .llcrypt registered [ 696.677268] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 19:00:53 (1763337653) [ 696.965675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 696.983218] alg: No test for adler32 (adler32-zlib) [ 697.918964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.960045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 697.999069] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 698.702144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.721926] LNet: 41600:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.730263] LNetError: 41600:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.747300] LNet: Removed LNI 192.168.204.14@tcp [ 698.771955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 698.832390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 698.861980] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 705.393933] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 19:01:01 (1763337661) [ 705.495316] LNet: 42194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.521229] LNetError: 42194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 705.568510] LNet: Removed LNI 192.168.204.14@tcp [ 706.232535] Key type .llcrypt unregistered [ 706.234413] Key type ._llcrypt unregistered [ 711.438457] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 711.530526] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 711.540697] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 711.545741] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 714.066091] Key type ._llcrypt registered [ 714.067837] Key type .llcrypt registered [ 714.303925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 714.339461] alg: No test for adler32 (adler32-zlib) [ 715.238582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.307637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.355787] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 716.135526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 716.151843] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.247778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.276417] LNet: 42938:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.282327] LNetError: 42938:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.288750] LNet: Removed LNI 192.168.204.14@tcp [ 718.308127] LNet: Removed LNI 10.1.2.3@tcp [ 718.341630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.400152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 718.436257] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 718.440119] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 718.892474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.937936] LNet: 42965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.941954] LNetError: 42965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.947622] LNetError: 42965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 718.952681] LNet: Removed LNI 192.168.204.14@tcp [ 718.973674] LNet: Removed LNI 10.1.2.3@tcp [ 719.004629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.115211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 719.170635] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 719.179759] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.745827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 719.767593] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 720.744319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.758529] LNet: 43127:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.764803] LNetError: 43127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 720.768283] LNetError: 43127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 720.787705] LNet: Removed LNI 192.168.204.14@tcp [ 720.793989] LNet: Removed LNI 10.1.2.3@tcp [ 720.804744] LNet: Removed LNI 192.168.204.14@tcp2 [ 720.834691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.911938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 720.951182] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 720.959157] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 720.970792] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 727.894485] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 19:01:24 (1763337684) [ 728.074100] LNet: 43725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 728.083634] LNetError: 43725:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 728.087314] LNetError: 43725:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 728.099281] LNet: Removed LNI 192.168.204.14@tcp [ 728.104719] LNet: Removed LNI 10.1.2.3@tcp [ 728.119529] LNet: Removed LNI 192.168.204.14@tcp2 [ 728.738412] Key type .llcrypt unregistered [ 728.740635] Key type ._llcrypt unregistered [ 734.372466] Key type ._llcrypt registered [ 734.374365] Key type .llcrypt registered [ 734.602438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 734.615346] alg: No test for adler32 (adler32-zlib) [ 735.595913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.705894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.746837] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 736.657531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.687692] LNet: 44116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.698478] LNetError: 44116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 736.713945] LNet: Removed LNI 192.168.204.14@tcp [ 736.742363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.804689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 736.876035] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 737.692123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.713429] LNet: 44146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.725837] LNetError: 44146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.743379] LNet: Removed LNI 192.168.204.14@tcp [ 737.777683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.869870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 737.892837] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 738.300547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.330771] LNet: Removed LNI 192.168.204.14@tcp [ 738.364763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.404641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 738.432489] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 738.727135] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.833884] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.837040] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.840533] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.220846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.237698] LNet: 44251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.242056] LNet: 44251:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 741.246703] LNetError: 44251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.249488] LNetError: 44251:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 741.263594] LNet: Removed LNI 192.168.204.14@tcp [ 741.282558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.372411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 741.398286] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 748.518617] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 19:01:44 (1763337704) [ 748.604459] LNet: 44839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.615962] LNetError: 44839:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.646055] LNet: Removed LNI 192.168.204.14@tcp [ 749.492365] Key type .llcrypt unregistered [ 749.495026] Key type ._llcrypt unregistered [ 754.659509] Key type ._llcrypt registered [ 754.660977] Key type .llcrypt registered [ 754.966862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 754.989061] alg: No test for adler32 (adler32-zlib) [ 755.957159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.044721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.078687] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 757.085627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.107525] LNet: 45240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.112974] LNetError: 45240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.130204] LNet: Removed LNI 192.168.204.14@tcp [ 757.153442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 757.242868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 757.293811] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 763.733631] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 19:02:00 (1763337720) [ 763.821378] LNet: 45823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.832197] LNetError: 45823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.855864] LNet: Removed LNI 192.168.204.14@tcp [ 764.395405] Key type .llcrypt unregistered [ 764.397373] Key type ._llcrypt unregistered [ 769.071971] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.179839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.184708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.187957] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 771.625168] Key type ._llcrypt registered [ 771.626918] Key type .llcrypt registered [ 771.815314] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 771.839903] alg: No test for adler32 (adler32-zlib) [ 772.804267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 772.888780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 772.932264] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 773.768377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 773.782446] LNet: 46276:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 773.789648] LNetError: 46276:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 773.802795] LNet: Removed LNI 192.168.204.14@tcp [ 773.865693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.923158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 773.950691] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 779.371792] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:02:16 (1763337736) [ 779.534608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.567380] LNet: 46875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.571626] LNetError: 46875:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.583248] LNet: Removed LNI 192.168.204.14@tcp [ 779.600678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.678921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 785.031857] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:02:21 (1763337741) [ 785.220625] LNet: 47456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.779247] Key type .llcrypt unregistered [ 785.780992] Key type ._llcrypt unregistered [ 790.763276] Key type ._llcrypt registered [ 790.767545] Key type .llcrypt registered [ 790.992785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 791.001550] alg: No test for adler32 (adler32-zlib) [ 791.862852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 791.909239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 791.929817] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 792.645042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 792.658529] LNet: 47850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.663474] LNetError: 47850:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 792.672319] LNet: Removed LNI 192.168.204.14@tcp [ 792.694654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 792.757573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 792.790354] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 798.547987] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:02:35 (1763337755) [ 798.627491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.668619] LNet: 48436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 798.681296] LNetError: 48436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 798.706177] LNet: Removed LNI 192.168.204.14@tcp [ 798.788404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.871454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 804.710061] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:02:41 (1763337761) [ 806.094271] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 807.775525] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:02:44 (1763337764) [ 809.242499] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 810.826172] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:02:47 (1763337767) [ 812.323621] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 813.686840] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:02:50 (1763337770) [ 814.960940] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 816.491401] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:02:53 (1763337773) [ 818.064464] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 819.567787] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:02:56 (1763337776) [ 819.844975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.862347] LNet: 50495:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.888591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.937766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 819.962555] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 820.685761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.14 [ 820.708759] LNet: Removed LNI 192.168.204.14@tcp [ 826.652128] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:03:03 (1763337783) [ 826.905741] LNet: 51198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.911803] LNetError: 51198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.936696] LNet: Removed LNI 192.168.204.14@tcp [ 827.689952] Key type .llcrypt unregistered [ 827.692718] Key type ._llcrypt unregistered [ 832.888839] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 833.081980] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 833.090404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 833.096420] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.777850] Key type ._llcrypt registered [ 835.779980] Key type .llcrypt registered [ 835.959036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.982401] alg: No test for adler32 (adler32-zlib) [ 837.360441] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 843.600706] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:03:20 (1763337800) [ 843.808352] LNet: 52099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 843.819624] LNetError: 52099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 843.840952] LNet: Removed LNI 10.1.2.3@tcp [ 844.387417] Key type .llcrypt unregistered [ 844.395324] Key type ._llcrypt unregistered [ 849.563494] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 849.750322] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 849.755481] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 849.765377] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.364824] Key type ._llcrypt registered [ 852.366469] Key type .llcrypt registered [ 852.601617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 852.623779] alg: No test for adler32 (adler32-zlib) [ 853.525602] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 859.759220] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:03:36 (1763337816) [ 859.874712] LNet: 52979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 859.885367] LNetError: 52979:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 859.899024] LNet: Removed LNI 10.1.2.3@tcp [ 860.502613] Key type .llcrypt unregistered [ 860.506904] Key type ._llcrypt unregistered [ 865.546738] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 865.732421] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 865.736601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 865.746168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 868.510885] Key type ._llcrypt registered [ 868.514716] Key type .llcrypt registered [ 868.697505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.720337] alg: No test for adler32 (adler32-zlib) [ 869.668890] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 876.084858] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:03:52 (1763337832) [ 876.288794] LNet: 53874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 876.305991] LNetError: 53874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 876.346404] LNet: Removed LNI 10.1.2.3@tcp [ 877.148638] Key type .llcrypt unregistered [ 877.149973] Key type ._llcrypt unregistered [ 883.015377] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 883.274173] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 883.280744] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 883.282865] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 885.950056] Key type ._llcrypt registered [ 885.951417] Key type .llcrypt registered [ 886.187962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 886.211261] alg: No test for adler32 (adler32-zlib) [ 887.176520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 887.237889] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 895.284977] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:04:11 (1763337851) [ 895.703077] LNet: 55103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.709586] LNetError: 55103:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 895.727439] LNet: Removed LNI 10.1.2.3@tcp [ 896.292308] Key type .llcrypt unregistered [ 896.294076] Key type ._llcrypt unregistered [ 901.648651] Key type ._llcrypt registered [ 901.650216] Key type .llcrypt registered [ 901.832102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.844624] alg: No test for adler32 (adler32-zlib) [ 902.849171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 902.887721] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 909.349242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 910.009353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_interrupt -r 1 [ 910.088819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_interrupt -r 1 [ 910.142636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 910.167452] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 910.426740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 910.494396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 910.543317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 910.595801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 910.644823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 910.686563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 910.727546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 910.768181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 910.806315] LNet: 55536:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 910.823618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 910.957251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 910.977653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 911.012799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 911.052715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 911.074619] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 911.084645] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 911.096988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 911.140237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 911.569779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_dropped -r 1 [ 911.612696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_dropped -r 1 [ 911.654374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 911.841837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 911.875894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 911.915692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 911.961734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 911.992520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 912.018117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 912.045383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 912.076924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 912.095240] LNet: 55645:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 912.115398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 912.140272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 912.177809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 912.210592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 912.243132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 912.282984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 912.319602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 912.777472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_aborted -r 1 [ 912.833946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_aborted -r 1 [ 912.899696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 912.947570] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 912.950986] LNet: Skipped 1 previous similar message [ 913.209593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 913.240177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 913.275752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 913.306746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 913.338765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 913.375193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 913.423683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 913.462451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 913.480322] LNet: 55757:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 913.506957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 913.556534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 913.607718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 913.653548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 913.700977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 913.736706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 913.767142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 914.283674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_no_route -r 1 [ 914.318905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_no_route -r 1 [ 914.356919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 914.573058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 914.603598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 914.637987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 914.696371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 914.754705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 914.784213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 914.818378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 914.844611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 914.879593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 914.919911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 914.952728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 914.988331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 915.013357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 915.047354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 915.082479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 915.484248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_timeout -r 1 [ 915.515271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_timeout -r 1 [ 915.534751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 915.554695] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 915.558686] LNet: Skipped 2 previous similar messages [ 915.754643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 915.778658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 915.802173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 915.826747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 915.850907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 915.886898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 915.929869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 915.952380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 915.962191] LNet: 55975:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 915.967238] LNet: 55975:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 915.980138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 916.017574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 916.050241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 916.092803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 916.133822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 916.195101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 916.225726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 916.774233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_error -r 1 [ 916.799186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_error -r 1 [ 916.836193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 917.044360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 917.092974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 917.130730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 917.157518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 917.196660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 917.229071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 917.253122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 917.289652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 917.322443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 917.364902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 917.390792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 917.419446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.448130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 917.487226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 917.523353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 917.731431] LNet: 56125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.737393] LNetError: 56125:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 917.749641] LNet: Removed LNI 192.168.204.14@tcp [ 918.067430] Key type .llcrypt unregistered [ 918.069093] Key type ._llcrypt unregistered [ 932.896660] Key type ._llcrypt registered [ 932.899214] Key type .llcrypt registered [ 933.862600] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:04:50 (1763337890) [ 934.554536] Key type .llcrypt unregistered [ 934.556236] Key type ._llcrypt unregistered [ 938.909992] Key type ._llcrypt registered [ 938.911723] Key type .llcrypt registered [ 939.161308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 939.187439] alg: No test for adler32 (adler32-zlib) [ 940.136664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 940.198391] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 946.195360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 952.930061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 952.966721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 952.999455] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 954.660725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_interrupt -r 1 [ 954.692562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_interrupt -r 1 [ 954.722757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_interrupt -r 1 [ 954.763595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_interrupt -r 1 [ 954.807404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_interrupt -r 1 [ 954.851608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_interrupt -r 1 [ 954.903138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_interrupt -r 1 [ 954.953112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_interrupt -r 1 [ 955.015824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 955.054946] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 955.059307] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 955.376882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 955.418676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 955.454872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 955.493807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 955.528533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 955.552577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 955.572938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 955.610084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 955.623890] LNet: 57584:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 955.641052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 955.674588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 955.699535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.730251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 955.762589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.811030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.865956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 956.549467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_dropped -r 1 [ 956.597805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_dropped -r 1 [ 956.643501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_dropped -r 1 [ 956.690585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_dropped -r 1 [ 956.728713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_dropped -r 1 [ 956.776302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_dropped -r 1 [ 956.810371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_dropped -r 1 [ 956.857092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_dropped -r 1 [ 956.917317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 956.940337] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 956.943876] LNet: Skipped 2 previous similar messages [ 957.212962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 957.264376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 957.312970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 957.347736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 957.381521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 957.404466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 957.436384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 957.480099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 957.498845] LNet: 57705:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 957.513595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 957.555710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 957.584416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 957.624894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 957.671686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 957.713743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 957.732670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 958.147398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_aborted -r 1 [ 958.176751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_aborted -r 1 [ 958.222430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_aborted -r 1 [ 958.256851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_aborted -r 1 [ 958.303357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_aborted -r 1 [ 958.345836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_aborted -r 1 [ 958.418994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_aborted -r 1 [ 958.461754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_aborted -r 1 [ 958.493556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 958.513663] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 958.517944] LNet: Skipped 2 previous similar messages [ 958.672211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 958.704260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 958.735255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 958.770424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 958.798568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 958.835773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 958.858171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 958.891095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 958.912881] LNet: 57827:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 958.945664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 959.000862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 959.031175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 959.048770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 959.080368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 959.108256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 959.137416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 959.551306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_no_route -r 1 [ 959.581697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_no_route -r 1 [ 959.610404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_no_route -r 1 [ 959.641871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_no_route -r 1 [ 959.681436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_no_route -r 1 [ 959.716830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_no_route -r 1 [ 959.738874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_no_route -r 1 [ 959.800763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_no_route -r 1 [ 959.850325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 960.071543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 960.096769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 960.127146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 960.164910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 960.188873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 960.212506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 960.244061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 960.267683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 960.288295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 960.315624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 960.339654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 960.363850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 960.391362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 960.415773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 960.451868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 960.849946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_timeout -r 1 [ 960.889719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_timeout -r 1 [ 960.919404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_timeout -r 1 [ 960.944587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_timeout -r 1 [ 960.974501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_timeout -r 1 [ 961.005928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_timeout -r 1 [ 961.032642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_timeout -r 1 [ 961.058388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_timeout -r 1 [ 961.087648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 961.107247] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 961.112843] LNet: Skipped 5 previous similar messages [ 961.339723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 961.360259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 961.383410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 961.404774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 961.435962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 961.468791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 961.492552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 961.524754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 961.544256] LNet: 58069:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 961.550465] LNet: 58069:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 961.566259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 961.597391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 961.635806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 961.670539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 961.707383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 961.741880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 961.774710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 962.229463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_error -r 1 [ 962.257648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_error -r 1 [ 962.303208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_error -r 1 [ 962.329038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_error -r 1 [ 962.352213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_error -r 1 [ 962.385185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_error -r 1 [ 962.421291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_error -r 1 [ 962.451535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_error -r 1 [ 962.482170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 962.674408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 962.713892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 962.745181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 962.778417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 962.808800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 962.843519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 962.865935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 962.897298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 962.932711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 962.961539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 962.995938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 963.025349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 963.061801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 963.092475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 963.130896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 967.563845] LNet: 58265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 967.571178] LNetError: 58265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.607173] LNet: Removed LNI 192.168.204.14@tcp [ 968.611650] LNetError: 58265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 968.631730] LNet: Removed LNI 192.168.204.14@tcp1 [ 969.102782] Key type .llcrypt unregistered [ 969.104617] Key type ._llcrypt unregistered [ 983.545688] Key type ._llcrypt registered [ 983.547434] Key type .llcrypt registered [ 984.753493] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:05:41 (1763337941) [ 985.724444] Key type .llcrypt unregistered [ 985.729802] Key type ._llcrypt unregistered [ 990.722403] Key type ._llcrypt registered [ 990.724743] Key type .llcrypt registered [ 990.925198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.938569] alg: No test for adler32 (adler32-zlib) [ 991.898279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 991.943185] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 998.579830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 999.418112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_dropped -r 1 [ 999.471202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_dropped -r 1 [ 999.511693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 999.539105] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 999.858862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 999.927332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 999.989810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1000.042883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1000.076279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1000.122828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1000.172788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1000.228462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1000.244335] LNet: 59508:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1000.273947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1000.312743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1000.354602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1000.397192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1000.431547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1000.506069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1000.598859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1001.289489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_error -r 1 [ 1001.331811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_error -r 1 [ 1001.364899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 1001.414226] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1001.633040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1001.670497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1001.694625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1001.735753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1001.761060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1001.805978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1001.834531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1001.865282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1001.875980] LNet: 59618:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1001.907471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1001.948598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1001.999265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1002.049503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1002.100043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1002.136578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1002.191475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1002.857509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_timeout -r 1 [ 1002.909844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_timeout -r 1 [ 1002.966716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 1002.996061] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1003.182116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1003.210141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1003.245637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1003.270294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1003.295602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1003.325097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1003.363788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1003.396292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1003.411606] LNet: 59727:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1003.421879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1003.480154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1003.523559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1003.553545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1003.583784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1003.609080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1003.636625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1003.952494] LNet: 59776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.961753] LNetError: 59776:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.986584] LNet: Removed LNI 192.168.204.14@tcp [ 1004.459464] Key type .llcrypt unregistered [ 1004.463239] Key type ._llcrypt unregistered [ 1018.502516] Key type ._llcrypt registered [ 1018.515166] Key type .llcrypt registered [ 1019.582913] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:06:16 (1763337976) [ 1020.325044] Key type .llcrypt unregistered [ 1020.327294] Key type ._llcrypt unregistered [ 1025.022617] Key type ._llcrypt registered [ 1025.024533] Key type .llcrypt registered [ 1025.319705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1025.349290] alg: No test for adler32 (adler32-zlib) [ 1026.288296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1026.323447] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1032.942591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1040.622791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1040.691164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1040.718474] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1042.425323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_dropped -r 1 [ 1042.465159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e remote_dropped -r 1 [ 1042.505915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_dropped -r 1 [ 1042.571435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e remote_dropped -r 1 [ 1042.629358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e remote_dropped -r 1 [ 1042.681045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e remote_dropped -r 1 [ 1042.747381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e remote_dropped -r 1 [ 1042.793993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e remote_dropped -r 1 [ 1042.836764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 1042.873126] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1042.877072] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.114@tcp [ 1043.134747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1043.159371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1043.198328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1043.258774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1043.303949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1043.359160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1043.389867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1043.447173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1043.495189] LNet: 61239:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1043.543398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1043.591825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1043.611342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1043.635429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1043.654585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1043.674892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1043.692795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1043.722253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1044.184279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_error -r 1 [ 1044.212578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e remote_error -r 1 [ 1044.252566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_error -r 1 [ 1044.288655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e remote_error -r 1 [ 1044.317327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e remote_error -r 1 [ 1044.356992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e remote_error -r 1 [ 1044.387812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e remote_error -r 1 [ 1044.419295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e remote_error -r 1 [ 1044.470656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 1044.487620] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1044.490965] LNet: Skipped 2 previous similar messages [ 1044.744971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1044.774887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1044.822308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1044.864634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1044.896934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1044.934578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1044.989993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1045.031351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1045.048223] LNet: 61362:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1045.072883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1045.103971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1045.134522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1045.168771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1045.193156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1045.234498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1045.289074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1045.339971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1046.044647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_timeout -r 1 [ 1046.096247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e remote_timeout -r 1 [ 1046.140825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_timeout -r 1 [ 1046.167315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e remote_timeout -r 1 [ 1046.219571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e remote_timeout -r 1 [ 1046.270658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e remote_timeout -r 1 [ 1046.309251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e remote_timeout -r 1 [ 1046.347379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e remote_timeout -r 1 [ 1046.393500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.114@tcp [ 1046.423744] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1046.621362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1046.658642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1046.686558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1046.717968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1046.752712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1046.788412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1046.819765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1046.861119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1046.897398] LNet: 61486:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1046.939398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1046.977616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1047.023894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1047.060262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1047.083353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1047.142832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1047.195590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1047.256804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1052.053995] LNet: 61563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1052.062855] LNetError: 61563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1052.077754] LNet: Removed LNI 192.168.204.14@tcp [ 1052.091109] LNet: Removed LNI 192.168.204.14@tcp1 [ 1052.592682] Key type .llcrypt unregistered [ 1052.594433] Key type ._llcrypt unregistered [ 1067.104143] Key type ._llcrypt registered [ 1067.106121] Key type .llcrypt registered [ 1068.288463] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:07:04 (1763338024) [ 1068.792277] Key type .llcrypt unregistered [ 1068.793536] Key type ._llcrypt unregistered [ 1074.094815] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1074.154567] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1074.157324] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1074.159857] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1076.872716] Key type ._llcrypt registered [ 1076.876512] Key type .llcrypt registered [ 1077.018251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.032137] alg: No test for adler32 (adler32-zlib) [ 1077.967729] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1077.991968] LNet: 62647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.998639] LNetError: 62647:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1078.011755] LNet: Removed LNI 192.168.204.14@tcp [ 1078.586386] Key type .llcrypt unregistered [ 1078.588155] Key type ._llcrypt unregistered [ 1083.146562] Key type ._llcrypt registered [ 1083.148158] Key type .llcrypt registered [ 1083.387967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1083.412277] alg: No test for adler32 (adler32-zlib) [ 1084.326647] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1084.328919] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1084.381874] LNet: 62916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.391734] LNetError: 62916:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.401636] LNet: Removed LNI 192.168.204.14@tcp [ 1084.414930] LNet: Removed LNI 10.1.2.3@tcp1 [ 1085.099641] Key type .llcrypt unregistered [ 1085.101838] Key type ._llcrypt unregistered [ 1090.087369] Key type ._llcrypt registered [ 1090.088912] Key type .llcrypt registered [ 1090.260166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.276327] alg: No test for adler32 (adler32-zlib) [ 1091.286198] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1091.288561] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1091.321824] LNet: 63185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1091.327172] LNetError: 63185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1091.330889] LNet: Removed LNI 192.168.204.14@tcp [ 1091.341977] LNet: Removed LNI 10.1.2.3@tcp [ 1091.960410] Key type .llcrypt unregistered [ 1091.963473] Key type ._llcrypt unregistered [ 1097.647218] Key type ._llcrypt registered [ 1097.655569] Key type .llcrypt registered [ 1097.812904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1097.827480] alg: No test for adler32 (adler32-zlib) [ 1098.779171] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1098.781396] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.833070] LNet: 63456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.838525] LNetError: 63456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.846781] LNet: Removed LNI 192.168.204.14@tcp [ 1098.860211] LNet: Removed LNI 10.1.2.3@tcp [ 1099.403574] Key type .llcrypt unregistered [ 1099.409142] Key type ._llcrypt unregistered [ 1104.686673] Key type ._llcrypt registered [ 1104.688376] Key type .llcrypt registered [ 1104.865608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1104.879420] alg: No test for adler32 (adler32-zlib) [ 1105.860505] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1105.862600] LNetError: 63716:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1105.871518] LNetError: Error -100 starting up LNI tcp [ 1105.888182] LNet: Removed LNI 192.168.204.14@tcp [ 1106.514606] Key type .llcrypt unregistered [ 1106.516377] Key type ._llcrypt unregistered [ 1115.659857] Key type ._llcrypt registered [ 1115.661374] Key type .llcrypt registered [ 1117.178928] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:07:53 (1763338073) [ 1118.033442] Key type .llcrypt unregistered [ 1118.037174] Key type ._llcrypt unregistered [ 1123.121196] Key type ._llcrypt registered [ 1123.122661] Key type .llcrypt registered [ 1123.322428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.349094] alg: No test for adler32 (adler32-zlib) [ 1124.270458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.322764] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1131.397750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1132.109609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e network_timeout -r 1 [ 1132.141256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e network_timeout -r 1 [ 1132.186985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1132.215268] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1132.514429] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 1132.531528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.573426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1132.590291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.621839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.658151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.686508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.713198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.737225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.759556] LNet: 64966:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.770431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.790485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.836963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.881542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.926909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.954850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1133.132901] LNet: 65004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.136419] LNetError: 65004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1133.154148] LNet: Removed LNI 192.168.204.14@tcp [ 1133.579561] Key type .llcrypt unregistered [ 1133.581450] Key type ._llcrypt unregistered [ 1148.509110] Key type ._llcrypt registered [ 1148.510695] Key type .llcrypt registered [ 1148.754703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.790303] alg: No test for adler32 (adler32-zlib) [ 1149.687701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1149.714778] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1155.320520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1161.568922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1161.625563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1161.656542] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1163.128850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e network_timeout -r 1 [ 1163.161939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e network_timeout -r 1 [ 1163.186978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e network_timeout -r 1 [ 1163.211681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e network_timeout -r 1 [ 1163.251921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e network_timeout -r 1 [ 1163.281831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e network_timeout -r 1 [ 1163.309834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e network_timeout -r 1 [ 1163.342593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e network_timeout -r 1 [ 1163.372449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1163.397350] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1163.609289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.653079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1163.677542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.701755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.732649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.773879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.803488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.835268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.850234] LNet: 65877:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.868036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.890857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.913412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.940687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.972600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.999955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.582323] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1167.984803] LNet: 65949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.991186] LNetError: 65949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.000266] LNet: Removed LNI 192.168.204.14@tcp [ 1168.019591] LNet: Removed LNI 192.168.204.14@tcp1 [ 1168.411273] Key type .llcrypt unregistered [ 1168.413126] Key type ._llcrypt unregistered [ 1179.739145] Key type ._llcrypt registered [ 1179.740676] Key type .llcrypt registered [ 1181.088103] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:08:57 (1763338137) [ 1181.394271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.420069] alg: No test for adler32 (adler32-zlib) [ 1182.347624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.386176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1182.429858] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1183.014877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1183.030467] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1183.727844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1183.768895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1183.831210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1183.866484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1183.933961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1183.978021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1184.005362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.14@tcp [ 1184.025294] LNetError: 67046:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.14@tcp to recovery [ 1184.417647] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 1184.420747] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1186.463492] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1187.547675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1190.559403] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1191.192957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1191.258768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1191.335631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1191.355268] LNet: 67153:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.369622] LNetError: 67153:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.374868] LNet: Removed LNI 192.168.204.14@tcp [ 1191.389276] LNet: Removed LNI 192.168.204.14@tcp1 [ 1191.412764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.471361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1191.509243] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1192.272134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1192.297128] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1193.138851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1193.201338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.271404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1193.340280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1193.392606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1193.433965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1193.472696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1193.501628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.14@tcp [ 1193.520410] LNetError: 67425:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.14@tcp to recovery [ 1194.527749] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1197.096610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1200.672377] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1200.684794] LNet: Skipped 1 previous similar message [ 1205.241624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1205.322306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1205.369757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1212.798450] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:09:29 (1763338169) [ 1212.875540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1212.900847] LNet: 68136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.907192] LNetError: 68136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.915617] LNetError: 68136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1212.933648] LNet: Removed LNI 192.168.204.14@tcp [ 1212.967926] LNet: Removed LNI 192.168.204.14@tcp1 [ 1212.995817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1213.048190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1213.095246] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1213.842611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1213.855123] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1214.582473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1214.629947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1214.663410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1214.698237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1214.729595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1214.757315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1214.788713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.14@tcp --health 0 [ 1214.908285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1215.071359] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.14@tcp [ 1215.071458] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1215.077720] LNet: Skipped 1 previous similar message [ 1230.363419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1230.485350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1230.520737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.564158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.14@tcp --health 500 [ 1230.682098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1237.339357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1237.414874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1237.434885] LNet: 68635:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1237.447746] LNetError: 68635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1237.457798] LNetError: 68635:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1237.462226] LNet: Removed LNI 192.168.204.14@tcp [ 1237.479729] LNet: Removed LNI 192.168.204.14@tcp1 [ 1237.511896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1237.635744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1237.667786] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1238.383750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1238.405888] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1239.217135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1239.257712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.314809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1239.349365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1239.373552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1239.398651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1239.427768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1239.455494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.14@tcp --health 0 [ 1240.671466] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1240.685065] LNet: Skipped 3 previous similar messages [ 1241.850281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1252.025264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1252.085769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1252.152671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1258.847693] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:10:15 (1763338215) [ 1259.169159] LNet: 69692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1259.174946] LNetError: 69692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1259.179695] LNetError: 69692:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1259.183944] LNet: Removed LNI 192.168.204.14@tcp [ 1259.195672] LNet: Removed LNI 192.168.204.14@tcp1 [ 1259.685817] Key type .llcrypt unregistered [ 1259.687454] Key type ._llcrypt unregistered [ 1264.204244] Key type ._llcrypt registered [ 1264.207239] Key type .llcrypt registered [ 1264.392826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.412568] alg: No test for adler32 (adler32-zlib) [ 1265.340840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1265.371115] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1272.916482] Lustre: DEBUG MARKER: Initial discovery [ 1273.145587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.114@tcp [ 1275.081843] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1275.576798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1275.642538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.114@tcp [ 1275.702831] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1275.723393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.755840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.114@tcp -r 1 -m GET -l 3 [ 1275.803411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.114@tcp -r 1 -m GET -e local_error [ 1275.864222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.114@tcp -r 1 -m PUT -l 6 [ 1276.642380] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 1277.494575] Lustre: DEBUG MARKER: Force oleg414-client.virtnet to discover 192.168.204.114@tcp (in background) [ 1277.755164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.114@tcp [ 1278.943270] Lustre: DEBUG MARKER: Force oleg414-server to discover 192.168.204.14@tcp [ 1280.872825] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1281.320929] Lustre: DEBUG MARKER: Wait for 70451 [ 1282.925164] Lustre: DEBUG MARKER: Finished wait on 70451 [ 1283.160531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1283.233680] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1283.237393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1283.540168] LNet: 70899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1283.548491] LNetError: 70899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1283.577581] LNet: Removed LNI 192.168.204.14@tcp [ 1283.911685] Key type .llcrypt unregistered [ 1283.913360] Key type ._llcrypt unregistered [ 1298.495405] Key type ._llcrypt registered [ 1298.505366] Key type .llcrypt registered [ 1299.879774] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:10:56 (1763338256) [ 1300.479421] Key type .llcrypt unregistered [ 1300.481139] Key type ._llcrypt unregistered [ 1304.822442] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1304.942355] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1304.949251] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1304.956812] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1307.571077] Key type ._llcrypt registered [ 1307.573024] Key type .llcrypt registered [ 1307.735308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1307.745356] alg: No test for adler32 (adler32-zlib) [ 1308.689146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1308.742418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1308.771476] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1309.473988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1309.511751] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1317.693360] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:11:14 (1763338274) [ 1317.797274] LNet: 72959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.804206] LNetError: 72959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.841731] LNet: Removed LNI 192.168.204.14@tcp [ 1317.917327] LNet: Removed LNI 10.1.2.3@tcp [ 1318.745437] Key type .llcrypt unregistered [ 1318.751312] Key type ._llcrypt unregistered [ 1324.166855] Key type ._llcrypt registered [ 1324.168500] Key type .llcrypt registered [ 1324.310944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1324.328085] alg: No test for adler32 (adler32-zlib) [ 1325.233530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.270421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1325.289378] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1326.024957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1326.042497] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1327.058774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.14@tcp [ 1328.159378] LNetError: Unexpected error -22 connecting to 192.168.204.14@tcp at host 192.168.204.14:988 [ 1333.472104] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:11:29 (1763338289) [ 1333.551768] LNet: 74085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1333.561905] LNetError: 74085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1333.569864] LNet: Removed LNI 192.168.204.14@tcp [ 1333.583244] LNet: Removed LNI 10.1.2.3@tcp [ 1334.210623] Key type .llcrypt unregistered [ 1334.212318] Key type ._llcrypt unregistered [ 1339.869682] Key type ._llcrypt registered [ 1339.871989] Key type .llcrypt registered [ 1340.039727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1340.054101] alg: No test for adler32 (adler32-zlib) [ 1340.965344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1341.005874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1341.062360] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1341.722718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1341.742419] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1342.677834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.14@tcp1 --nid 192.168.204.14@tcp2 [ 1346.553881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp1 192.168.204.14@tcp2 [ 1346.570721] LNetError: 74923:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.14@tcp1 to 192.168.204.14@tcp2 - Route table may be misconfigured [ 1346.585539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp2 192.168.204.14@tcp1 [ 1352.522035] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:11:49 (1763338309) [ 1352.721154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1352.762848] LNet: 75497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1352.776148] LNetError: 75497:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1352.789134] LNet: Removed LNI 192.168.204.14@tcp1 [ 1353.839026] LNet: Removed LNI 192.168.204.14@tcp2 [ 1353.895867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1353.981975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.010807] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1354.832962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1354.847894] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1355.907820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1355.984931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.14@tcp [ 1356.025659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp1 192.168.204.14@tcp1 [ 1356.057876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.14@tcp -d 192.168.204.14@tcp -e network_timeout [ 1356.095419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -e network_timeout [ 1356.162882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -e network_timeout [ 1356.215363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -e network_timeout [ 1356.286546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1356.309439] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1356.327369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1356.419954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1356.530679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1357.026382] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 1362.142093] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:11:58 (1763338318) [ 1362.335706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.352906] LNet: 76358:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1362.364140] LNetError: 76358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.367238] LNetError: 76358:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1362.378907] LNet: Removed LNI 192.168.204.14@tcp [ 1363.435037] LNet: Removed LNI 192.168.204.14@tcp1 [ 1363.470615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1363.596492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1363.891436] LNet: 76395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1364.236296] Key type .llcrypt unregistered [ 1364.238856] Key type ._llcrypt unregistered [ 1373.201264] Key type ._llcrypt registered [ 1373.202953] Key type .llcrypt registered [ 1374.225495] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:12:10 (1763338330) [ 1374.578981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1374.598236] alg: No test for adler32 (adler32-zlib) [ 1375.535789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.800313] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1378.569217] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:12:15 (1763338335) [ 1378.797223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.825811] LNet: 77490:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.852489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1378.917772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1378.964260] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1379.692798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1379.710795] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1380.445659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1380.491268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp1 [ 1380.521118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp1 [ 1387.665806] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:12:23 (1763338343) [ 1389.525140] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg414-server [ 1391.077863] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:12:27 (1763338347) [ 1392.824367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg414-server [ 1394.809611] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:12:31 (1763338351) [ 1396.546490] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg414-server [ 1398.782182] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:12:34 (1763338354) [ 1401.291794] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg414-server [ 1403.332723] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:12:39 (1763338359) [ 1405.102617] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg414-server [ 1407.042229] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:12:43 (1763338363) [ 1408.659637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg414-server [ 1410.783600] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:12:47 (1763338367) [ 1412.351409] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg414-server [ 1414.607710] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:12:50 (1763338370) [ 1416.303936] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg414-server [ 1418.144968] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:12:54 (1763338374) [ 1420.375959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg414-server [ 1422.481581] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:12:58 (1763338378) [ 1422.609802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.665900] LNet: 81363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.669873] LNetError: 81363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.711259] LNet: Removed LNI 192.168.204.14@tcp [ 1423.713617] LNetError: 81363:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1423.738209] LNet: Removed LNI 192.168.204.14@tcp1 [ 1423.781513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.908990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.947070] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1425.236530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1425.371560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1425.625733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.680736] LNet: 81517:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.698556] LNetError: 81517:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.720885] LNet: Removed LNI 192.168.204.14@tcp [ 1425.758674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.840943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1425.861152] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1426.613527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1426.824636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1427.879262] LNet: Removed LNI 192.168.204.14@tcp [ 1427.905257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1427.973599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.014810] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1428.890597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1428.983186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1429.224584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1429.258675] LNet: 81810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1429.264232] LNet: 81810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1429.270364] LNetError: 81810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1429.276286] LNetError: 81810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1429.288885] LNet: Removed LNI 192.168.204.14@tcp [ 1429.308099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.359940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1429.414683] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1430.237412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1430.516447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.592767] LNet: Removed LNI 192.168.204.14@tcp [ 1431.628212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.724732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1431.749439] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1432.590121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1432.688390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1432.937993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.021805] LNet: Removed LNI 192.168.204.14@tcp [ 1434.038535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1434.103641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1434.141064] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1435.122731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1435.367051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1435.408770] LNet: 82248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1435.414175] LNet: 82248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1435.431978] LNetError: 82248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1435.435187] LNetError: 82248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1435.478983] LNet: Removed LNI 192.168.204.14@tcp [ 1435.507143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.611247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1435.660661] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1436.570327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1436.672234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1437.012934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1438.123850] LNet: Removed LNI 192.168.204.14@tcp [ 1438.183632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1438.276295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1438.309763] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1439.049650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1440.422461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1441.509881] LNet: Removed LNI 192.168.204.14@tcp [ 1441.541330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.594503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1441.616946] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1442.342671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1442.419381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1442.791058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1443.883886] LNet: Removed LNI 192.168.204.14@tcp [ 1443.922027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.005436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1444.042133] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1444.844527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1446.510589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1446.547978] LNet: 82852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1446.553179] LNet: 82852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1446.568152] LNetError: 82852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1446.577895] LNetError: 82852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1446.602868] LNet: Removed LNI 192.168.204.14@tcp [ 1446.630170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.733689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.800739] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1447.817611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1447.933858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1449.471552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1450.539550] LNet: Removed LNI 192.168.204.14@tcp [ 1450.564530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1450.611451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1450.639114] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1451.560268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1453.016903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1454.117458] LNet: Removed LNI 192.168.204.14@tcp [ 1454.162446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1454.245472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1454.266696] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1454.990509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1455.046172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.094532] LNet: Removed LNI 192.168.204.14@tcp [ 1455.130468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.201539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.238372] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1455.945670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1462.760302] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:13:39 (1763338419) [ 1462.964434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1463.001447] LNet: 83998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.004329] LNet: 83998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1463.028251] LNetError: 83998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1463.031188] LNetError: 83998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1463.085102] LNet: Removed LNI 192.168.204.14@tcp [ 1463.162748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.220524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1463.247850] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/180] [ 1464.552325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1464.601935] LNet: Removed LNI 192.168.204.14@tcp231 [ 1464.631925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1464.737680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1464.784796] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/180] [ 1464.791646] LNet: 84145:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1465.290667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1465.322500] LNet: Removed LNI 192.168.204.14@tcp231 [ 1465.347530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1465.367945] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/180] [ 1466.345843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.370359] LNet: Removed LNI 192.168.204.14@tcp231 [ 1466.386523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.452681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1466.480305] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/0] [ 1466.486751] LNet: 84297:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1466.814551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1466.850230] LNet: Removed LNI 192.168.204.14@tcp231 [ 1466.862600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1466.901415] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/0] [ 1468.034410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.112984] LNet: Removed LNI 192.168.204.14@tcp231 [ 1468.156314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1468.277663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1468.307034] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/60] [ 1468.318833] LNet: 84449:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.692401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1468.731125] LNet: Removed LNI 192.168.204.14@tcp231 [ 1468.745097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1468.767710] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/60] [ 1469.860453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.885730] LNet: Removed LNI 192.168.204.14@tcp231 [ 1469.900970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.959811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1469.992048] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/180] [ 1470.340260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1470.366935] LNet: Removed LNI 192.168.204.14@tcp231 [ 1470.389926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1470.412288] LNet: Added LNI 192.168.204.14@tcp231 [8/256/0/180] [ 1477.006739] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:13:53 (1763338433) [ 1477.137929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1477.189872] LNet: Removed LNI 192.168.204.14@tcp231 [ 1477.217941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.312762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1477.342679] LNet: Added LNI 192.168.204.14@tcp232 [8/256/0/180] [ 1478.116211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1478.184719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1478.210113] LNet: Removed LNI 192.168.204.14@tcp232 [ 1478.236544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1478.282490] LNet: Added LNI 192.168.204.14@tcp232 [8/256/0/180] [ 1479.209599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1479.249831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1479.280070] LNet: Removed LNI 192.168.204.14@tcp232 [ 1479.307527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1479.339701] LNet: Added LNI 192.168.204.14@tcp232 [8/256/0/180] [ 1480.310815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1480.358605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1480.380313] LNet: Removed LNI 192.168.204.14@tcp232 [ 1480.397777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1480.453806] LNet: Added LNI 192.168.204.14@tcp232 [8/256/0/180] [ 1481.176189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1481.230964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1481.277315] LNet: Removed LNI 192.168.204.14@tcp232 [ 1481.300984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1488.751904] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:14:05 (1763338445) [ 1489.794097] Key type .llcrypt unregistered [ 1489.798484] Key type ._llcrypt unregistered [ 1495.358713] Key type ._llcrypt registered [ 1495.364927] Key type .llcrypt registered [ 1495.615247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.629533] alg: No test for adler32 (adler32-zlib) [ 1496.573177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1496.597257] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1503.269444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1509.863957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1509.906926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1509.928649] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1511.212413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_interrupt -r 2 [ 1511.250357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_interrupt -r 2 [ 1511.293511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_interrupt -r 2 [ 1511.336762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_interrupt -r 2 [ 1511.368334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_interrupt -r 2 [ 1511.419352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_interrupt -r 2 [ 1511.475822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_interrupt -r 2 [ 1511.540292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_interrupt -r 2 [ 1511.600605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.114@tcp [ 1511.634987] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1511.643458] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 1511.906625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1511.947684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1511.981339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1512.022297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1512.053217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1512.086898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1512.115425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1512.141733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1512.163456] LNet: 87096:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1512.180426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1512.206579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1512.267491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1512.298424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.326755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1512.359867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1512.878093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_dropped -r 2 [ 1512.910527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_dropped -r 2 [ 1512.953065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_dropped -r 2 [ 1512.987793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_dropped -r 2 [ 1513.011973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_dropped -r 2 [ 1513.047178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_dropped -r 2 [ 1513.078951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_dropped -r 2 [ 1513.111249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_dropped -r 2 [ 1513.146527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.114@tcp [ 1513.170472] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1513.380674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1513.412739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1513.462810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1513.485621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1513.513788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1513.546818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1513.572725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1513.601236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1513.620577] LNet: 87216:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1513.636678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1513.665149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1513.708260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1513.747395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1513.774982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1513.868936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1514.398360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_aborted -r 2 [ 1514.427527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_aborted -r 2 [ 1514.463302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_aborted -r 2 [ 1514.498333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_aborted -r 2 [ 1514.540379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_aborted -r 2 [ 1514.580248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_aborted -r 2 [ 1514.607662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_aborted -r 2 [ 1514.639147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_aborted -r 2 [ 1514.677569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.114@tcp [ 1514.703484] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1514.968445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1515.005973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1515.046054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1515.077867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1515.105660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1515.136980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1515.175670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1515.217518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1515.244886] LNet: 87336:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1515.264632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1515.290420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1515.319131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1515.344236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1515.386980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1515.419173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1515.890306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_no_route -r 2 [ 1515.922805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_no_route -r 2 [ 1515.972132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_no_route -r 2 [ 1516.012833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_no_route -r 2 [ 1516.074069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_no_route -r 2 [ 1516.124232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_no_route -r 2 [ 1516.169078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_no_route -r 2 [ 1516.193584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_no_route -r 2 [ 1516.247656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.114@tcp [ 1516.492199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1516.513951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1516.535588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1516.564722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1516.590759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1516.623780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1516.656921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1516.701419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1516.732095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1516.762847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1516.800813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1516.834214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1516.864758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1516.900953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1517.437099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e local_timeout -r 2 [ 1517.481538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e local_timeout -r 2 [ 1517.534970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e local_timeout -r 2 [ 1517.590918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e local_timeout -r 2 [ 1517.649679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e local_timeout -r 2 [ 1517.705710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e local_timeout -r 2 [ 1517.751481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e local_timeout -r 2 [ 1517.795433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e local_timeout -r 2 [ 1517.835356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.114@tcp [ 1517.890934] LNet: There was an unexpected network error while writing to 192.168.204.114: rc = -22 [ 1517.911211] LNet: Skipped 1 previous similar message [ 1518.199318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1518.228857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1518.262249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1518.317757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1518.368638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1518.415418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1518.439479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1518.486912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1518.511942] LNet: 87577:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1518.517583] LNet: 87577:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1518.555831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1518.611109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1518.659247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1518.725093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1518.768326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1518.810290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1519.490541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp -m GET -e remote_dropped -r 2 [ 1519.524203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.114@tcp1 -m GET -e remote_dropped -r 2 [ 1519.563603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp -m GET -e remote_dropped -r 2 [ 1519.608714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp -d 192.168.204.14@tcp1 -m GET -e remote_dropped -r 2 [ 1519.665719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp -m GET -e remote_dropped -r 2 [ 1519.756622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.114@tcp1 -m GET -e remote_dropped -r 2 [ 1519.823978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp -m GET -e remote_dropped -r 2 [ 1519.905104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.14@tcp1 -d 192.168.204.14@tcp1 -m GET -e remote_dropped -r 2 [ 1519.948273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.14@tcp 192.168.204.114@tcp [ 1519.997102] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.114@tcp [ 1520.276771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1520.318838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1520.357505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1520.445735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1520.476593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1520.527522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1520.584917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1520.682445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1520.729987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1520.762832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1520.821356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1520.880442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1520.940506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1521.012761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.243381] LNet: 87770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1526.262535] LNetError: 87770:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1526.275202] LNet: Removed LNI 192.168.204.14@tcp [ 1526.286677] LNet: Removed LNI 192.168.204.14@tcp1 [ 1526.860359] Key type .llcrypt unregistered [ 1526.863570] Key type ._llcrypt unregistered [ 1543.938111] Key type ._llcrypt registered [ 1543.940823] Key type .llcrypt registered [ 1545.678099] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:15:01 (1763338501) [ 1546.050184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1546.077190] alg: No test for adler32 (adler32-zlib) [ 1547.097303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.186242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1547.210611] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1548.308932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1548.340795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.14@tcp [ 1548.368456] LNetError: 88747:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.14@tcp to recovery [ 1548.496479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.14@tcp [ 1548.620302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1555.707701] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:15:12 (1763338512) [ 1555.868359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.921112] LNet: 89330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.932153] LNetError: 89330:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1555.964914] LNet: Removed LNI 192.168.204.14@tcp [ 1556.010451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.130954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1556.165606] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1556.912496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1556.936514] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1557.836525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.14@tcp [ 1557.861500] LNetError: 89603:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.14@tcp to recovery [ 1558.175454] LNet: 1 local NIs in recovery (showing 1): 192.168.204.14@tcp [ 1558.176849] LNet: There was an unexpected network error while writing to 192.168.204.14: rc = -22 [ 1559.171848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.14@tcp [ 1566.383551] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:15:23 (1763338523) [ 1567.793747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg414-server [ 1569.076688] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:15:26 (1763338526) [ 1569.215121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.232454] LNet: 90550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.237420] LNetError: 90550:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.241752] LNet: Removed LNI 192.168.204.14@tcp [ 1569.253763] LNet: Removed LNI 192.168.204.14@tcp1 [ 1569.273428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.324589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1569.346261] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1570.034972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1570.081746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1570.117756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1570.199419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1570.223819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1570.253343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1575.112327] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:15:31 (1763338531) [ 1575.186565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.204100] LNet: 91269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.210769] LNetError: 91269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.223459] LNetError: 91269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1575.251611] LNet: Removed LNI 192.168.204.14@tcp [ 1575.290793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.329763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.375206] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1581.350945] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:15:38 (1763338538) [ 1581.458591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.484074] LNet: 91974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.490917] LNetError: 91974:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.511553] LNet: Removed LNI 192.168.204.14@tcp [ 1581.547294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.628776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1581.655879] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1587.353477] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:15:44 (1763338544) [ 1588.798162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1590.668185] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:15:47 (1763338547) [ 1591.137644] LNet: 93013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.146995] LNetError: 93013:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.167549] LNet: Removed LNI 192.168.204.14@tcp [ 1591.771802] Key type .llcrypt unregistered [ 1591.773169] Key type ._llcrypt unregistered [ 1596.585822] Key type ._llcrypt registered [ 1596.587419] Key type .llcrypt registered [ 1596.817240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1596.832799] alg: No test for adler32 (adler32-zlib) [ 1597.801761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1597.834781] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1603.969081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1607.436353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.114@tcp [ 1607.802696] LNet: 93441:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1607.811797] LNetError: 93441:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.825587] LNet: Removed LNI 192.168.204.14@tcp [ 1608.153613] Key type .llcrypt unregistered [ 1608.156520] Key type ._llcrypt unregistered [ 1618.325807] Key type ._llcrypt registered [ 1618.327703] Key type .llcrypt registered [ 1619.598851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1621.209839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1622.629485] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:16:19 (1763338579) [ 1623.010399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1623.022191] alg: No test for adler32 (adler32-zlib) [ 1623.984636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1624.052339] LNet: 94530:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1624.755587] Key type .llcrypt unregistered [ 1624.757492] Key type ._llcrypt unregistered [ 1630.116230] Key type ._llcrypt registered [ 1630.117753] Key type .llcrypt registered [ 1630.254987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1630.271365] alg: No test for adler32 (adler32-zlib) [ 1631.226438] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1631.228786] LNet: 94782:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1631.276204] LNetError: 94794:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.16@tcp is being used as a gateway but routing feature is not turned on [ 1631.298948] LNetError: 94794:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.16@tcp has gone from up to down [ 1631.910920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.039298] LNetError: 94794:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.15@tcp has gone from up to down [ 1635.061808] LNetError: 94794:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 1635.075726] LNet: 94840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1635.081166] LNetError: 94840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1635.129813] LNet: Removed LNI 192.168.204.14@tcp [ 1635.165600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.254140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1635.290328] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1635.295398] LNet: 94848:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1635.314134] LNet: 94848:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1635.328183] LNetError: 94848:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.16@tcp is a gateway. Can not delete it [ 1635.340272] LNetError: 94848:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.204.16@tcp is a router. Can not be deleted [ 1635.353946] LNet: 94848:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1635.376060] LNetError: 94845:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.204.18@tcp is being used as a gateway but routing feature is not turned on [ 1635.394083] LNetError: 94845:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 1642.438917] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:16:38 (1763338598) [ 1644.817903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg414-server [ 1646.439984] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:16:43 (1763338603) [ 1648.590839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg414-server [ 1650.768981] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:16:46 (1763338606) [ 1652.276533] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 1654.135239] Key type lgssc registered [ 1655.293533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1673.871191] Key type lgssc unregistered [ 1674.215926] LNet: 97265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1674.222138] LNetError: 97265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1674.234534] LNet: Removed LNI 192.168.204.14@tcp [ 1674.828254] Key type .llcrypt unregistered [ 1674.831666] Key type ._llcrypt unregistered [ 1688.667497] Key type ._llcrypt registered [ 1688.669263] Key type .llcrypt registered [ 1689.726448] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:17:26 (1763338646) [ 1690.064261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1690.073927] alg: No test for adler32 (adler32-zlib) [ 1690.978925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.065654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1691.099345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1691.146268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1691.210863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1696.561503] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:17:33 (1763338653) [ 1696.669201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.709340] LNet: 98666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1696.726430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.778674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.801472] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1697.612501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.631815] LNet: 98814:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.636652] LNetError: 98814:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1697.655625] LNet: Removed LNI 192.168.204.14@tcp [ 1697.669677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.712262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1697.739255] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1698.172970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.201248] LNetError: 98842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1698.216270] LNet: Removed LNI 192.168.204.14@tcp [ 1698.232282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.288965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1698.335470] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1703.419558] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:17:40 (1763338660) [ 1703.708521] LNet: 99434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.712949] LNet: 99434:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1703.718839] LNetError: 99434:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.740975] LNet: Removed LNI 192.168.204.14@tcp [ 1704.317509] Key type .llcrypt unregistered [ 1704.319360] Key type ._llcrypt unregistered [ 1709.463775] Key type ._llcrypt registered [ 1709.467686] Key type .llcrypt registered [ 1709.753787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1709.779689] alg: No test for adler32 (adler32-zlib) [ 1710.710817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.787615] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1710.793482] LNet: *** cfs_fail_loc=e003, val=100*** [ 1711.647512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1711.668773] LNet: *** cfs_fail_loc=e003, val=100*** [ 1711.671648] LNet: Skipped 2 previous similar messages [ 1711.682143] Ping reply received with mapping: [ 1711.683561] found metadata reply NI[192.168.204.14@tcp] [ 1711.684879] 0e cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1711.688653] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1711.692611] 5a 5a 5a 5a 5a 5a 5a 5a [ 1711.858842] LNet: 99827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1711.864848] LNetError: 99827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.937400] LNet: Removed LNI 192.168.204.14@tcp [ 1713.801908] Key type .llcrypt unregistered [ 1713.803347] Key type ._llcrypt unregistered [ 1724.117616] Key type ._llcrypt registered [ 1724.119354] Key type .llcrypt registered [ 1725.526966] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:18:01 (1763338681) [ 1725.926705] Key type .llcrypt unregistered [ 1725.928656] Key type ._llcrypt unregistered [ 1730.463487] Key type ._llcrypt registered [ 1730.465122] Key type .llcrypt registered [ 1730.734528] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1730.745511] alg: No test for adler32 (adler32-zlib) [ 1731.874520] LNetError: 100865:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1732.234440] Key type .llcrypt unregistered [ 1732.236231] Key type ._llcrypt unregistered [ 1736.618759] Key type ._llcrypt registered [ 1736.624161] Key type .llcrypt registered [ 1737.690347] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:18:14 (1763338694) [ 1738.384684] Key type .llcrypt unregistered [ 1738.388419] Key type ._llcrypt unregistered [ 1743.541943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1743.550594] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1746.046359] Key type ._llcrypt registered [ 1746.051876] Key type .llcrypt registered [ 1746.309236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1746.329366] alg: No test for adler32 (adler32-zlib) [ 1747.253562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.313387] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1757.490789] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:18:34 (1763338714) [ 1757.721834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.749870] LNet: 102649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.755439] LNetError: 102649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1757.767935] LNet: Removed LNI 10.1.2.3@tcp [ 1757.785699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.627720] Key type .llcrypt unregistered [ 1758.631624] Key type ._llcrypt unregistered [ 1764.597906] Key type ._llcrypt registered [ 1764.600517] Key type .llcrypt registered [ 1764.836390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1764.848711] alg: No test for adler32 (adler32-zlib) [ 1765.881089] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1765.978328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1766.196474] LNet: 102933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.213600] LNetError: 102933:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.232901] LNet: Removed LNI 192.168.204.14@tcp [ 1767.101342] Key type .llcrypt unregistered [ 1767.104755] Key type ._llcrypt unregistered [ 1777.799718] Key type ._llcrypt registered [ 1777.801511] Key type .llcrypt registered [ 1779.397324] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:18:55 (1763338735) [ 1780.112476] Key type .llcrypt unregistered [ 1780.114265] Key type ._llcrypt unregistered [ 1785.343089] Key type ._llcrypt registered [ 1785.344666] Key type .llcrypt registered [ 1785.672962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1785.697596] alg: No test for adler32 (adler32-zlib) [ 1793.327878] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:19:09 (1763338749) [ 1793.439319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.494839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.562524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.611942] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1794.558391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1794.602169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1794.687536] LNet: Removed LNI 192.168.204.14@tcp [ 1795.021299] LNet: 104669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.477683] Key type .llcrypt unregistered [ 1795.479437] Key type ._llcrypt unregistered [ 1806.849692] Key type ._llcrypt registered [ 1806.850949] Key type .llcrypt registered [ 1808.529332] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:19:24 (1763338764) [ 1808.939505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1808.955615] alg: No test for adler32 (adler32-zlib) [ 1809.997431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1810.112264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1810.167443] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1811.117671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1811.154446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.14@tcp [ 1811.174655] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1811.177595] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1811.180659] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1811, 0, 0 [ 1811.183893] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1811.185978] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1811.188204] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1811.191416] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1811, 0, 0 [ 1811.194478] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1811.196422] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1811.199456] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1811.201769] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1811, 0, 1861 [ 1811.205843] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1811.208155] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1811.210670] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1811.214516] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1811, 0, 1861 [ 1811.219137] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1818.347852] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:19:34 (1763338774) [ 1818.707688] LNet: 106206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1818.716198] LNetError: 106206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1818.731649] LNet: Removed LNI 192.168.204.14@tcp [ 1819.287562] Key type .llcrypt unregistered [ 1819.289778] Key type ._llcrypt unregistered [ 1824.565268] Key type ._llcrypt registered [ 1824.566597] Key type .llcrypt registered [ 1824.906409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1824.934874] alg: No test for adler32 (adler32-zlib) [ 1825.940281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1825.982620] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1833.003775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 1840.500898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.566527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1840.588518] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 1841.806694] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1842.016970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1842.019249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1842.029017] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1844.481657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1844.517811] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1854.162518] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:20:10 (1763338810) [ 1854.320558] LNet: 107703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1854.339284] LNetError: 107703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1854.342935] LNet: Removed LNI 192.168.204.14@tcp [ 1855.391187] LNet: Removed LNI 192.168.204.14@tcp1 [ 1855.393167] LNetError: 107703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1855.396797] LNetError: 107703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1855.423799] LNet: Removed LNI 10.1.2.3@tcp99 [ 1856.198731] Key type .llcrypt unregistered [ 1856.200853] Key type ._llcrypt unregistered [ 1861.596837] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1861.599479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1864.218559] Key type ._llcrypt registered [ 1864.220808] Key type .llcrypt registered [ 1864.431746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1864.456692] alg: No test for adler32 (adler32-zlib) [ 1865.391609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1865.448193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1865.482671] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1866.392543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1866.413701] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1869.066723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1869.143599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1869.190795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1869.226689] LNetError: 108451:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1869.264146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1869.327984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1870.132851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1870.247803] LNet: 108471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.254887] LNetError: 108471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.263232] LNet: Removed LNI 192.168.204.14@tcp [ 1871.265348] LNetError: 108471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1871.291137] LNet: Removed LNI 10.1.2.3@tcp [ 1872.093693] Key type .llcrypt unregistered [ 1872.098832] Key type ._llcrypt unregistered [ 1883.032540] Key type ._llcrypt registered [ 1883.034206] Key type .llcrypt registered [ 1884.172807] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:20:40 (1763338840) [ 1884.667082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.694226] alg: No test for adler32 (adler32-zlib) [ 1885.614717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.711519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.730785] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1886.543943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.14@tcp [ 1886.621901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg414-client@tcp [ 1893.738081] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:20:50 (1763338850) [ 1893.959630] LNet: 110001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.972475] LNetError: 110001:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1895.014225] LNet: Removed LNI 192.168.204.14@tcp [ 1895.720501] Key type .llcrypt unregistered [ 1895.722075] Key type ._llcrypt unregistered [ 1901.585320] Key type ._llcrypt registered [ 1901.586913] Key type .llcrypt registered [ 1901.785247] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1901.799478] alg: No test for adler32 (adler32-zlib) [ 1902.727863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1902.753611] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1902.782789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1909.034913] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:21:05 (1763338865) [ 1910.504466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1912.837382] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:21:09 (1763338869) [ 1914.231861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1915.967277] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:21:12 (1763338872) [ 1916.203895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.248880] LNet: 111418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.263181] LNetError: 111418:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.277543] LNet: Removed LNI 192.168.204.14@tcp [ 1916.296078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.393942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1916.413968] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1917.217743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1917.303287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1924.275974] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:21:20 (1763338880) [ 1924.612595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1924.647043] LNet: 112128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.662318] LNetError: 112128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.692489] LNet: Removed LNI 192.168.204.14@tcp [ 1924.747167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1924.808032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1924.852308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1925.482307] Key type .llcrypt unregistered [ 1925.484300] Key type ._llcrypt unregistered [ 1935.840132] Key type ._llcrypt registered [ 1935.841842] Key type .llcrypt registered [ 1937.061071] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:21:33 (1763338893) [ 1937.422282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1937.460844] alg: No test for adler32 (adler32-zlib) [ 1938.342957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.406865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.464261] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1939.317900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1939.373737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1946.303824] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:21:42 (1763338902) [ 1946.647446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.681821] LNet: 113679:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.698551] LNetError: 113679:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.716349] LNet: Removed LNI 192.168.204.14@tcp [ 1946.746529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.817882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1946.847125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1953.407950] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:21:49 (1763338909) [ 1953.482357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.508732] LNet: 114250:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.533446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.634430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1954.264986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1954.315911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1954.385425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1954.887691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1954.919431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1955.022593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1955.041902] LNet: 114292:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.067549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.144813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1955.514546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1955.560530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1955.632919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1956.097549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1956.145613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1961.986935] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:21:58 (1763338918) [ 1962.121502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1962.145226] LNet: 114889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1962.160161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.263388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1962.304882] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1963.067747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1963.097503] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1963.949724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1964.007538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1964.259412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.273957] LNetError: 115172:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.278699] LNet: Removed LNI 192.168.204.14@tcp [ 1964.289876] LNet: Removed LNI 192.168.204.14@tcp2 [ 1964.320742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.395293] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1964.401586] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1964.404848] LNet: 115178:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1964.825278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1965.039826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1966.111890] LNet: Removed LNI 192.168.204.14@tcp [ 1966.118243] LNetError: 115215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.135784] LNetError: 115215:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1966.167248] LNet: Removed LNI 192.168.204.14@tcp2 [ 1966.203835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.262288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1966.293753] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1966.964129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1966.977577] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1967.760144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1967.791583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1967.847809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1968.140589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1968.154852] LNet: 115500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.158040] LNet: 115500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1968.173181] LNetError: 115500:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.181932] LNet: Removed LNI 192.168.204.14@tcp [ 1969.253519] LNet: Removed LNI 192.168.204.14@tcp2 [ 1969.294216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.351995] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1969.356386] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1969.362728] LNet: 115506:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1969.833253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1970.017733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1970.050084] LNet: Removed LNI 192.168.204.14@tcp [ 1971.109505] LNet: Removed LNI 192.168.204.14@tcp2 [ 1971.133412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1971.238903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1971.262716] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1971.992735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1972.030375] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1972.959810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1973.024781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1973.238365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1973.260445] LNetError: 115823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1973.263763] LNetError: 115823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1973.270233] LNet: Removed LNI 192.168.204.14@tcp [ 1973.288214] LNet: Removed LNI 192.168.204.14@tcp2 [ 1973.316792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.399105] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1973.404470] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1973.419490] LNet: 115829:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1974.018484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1974.231025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1975.263223] LNet: Removed LNI 192.168.204.14@tcp [ 1975.281556] LNet: Removed LNI 192.168.204.14@tcp2 [ 1975.302945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1975.364135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1975.390458] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1976.184480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1976.207960] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1977.180443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1977.228850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1977.273173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1977.471469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1977.496645] LNet: 116150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1977.502810] LNet: 116150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1978.530327] LNet: Removed LNI 192.168.204.14@tcp [ 1979.565141] LNet: Removed LNI 192.168.204.14@tcp2 [ 1979.623556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1979.722808] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1979.743674] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1979.754418] LNet: 116157:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1980.494236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1980.711476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1980.745776] LNet: Removed LNI 192.168.204.14@tcp [ 1981.807608] LNet: Removed LNI 192.168.204.14@tcp2 [ 1981.850175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1981.927228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1981.957093] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1983.243798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1983.271667] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1984.443673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1984.511671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1984.624404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1985.171666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1985.201765] LNetError: 116491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1985.219566] LNetError: 116491:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1985.237568] LNet: Removed LNI 192.168.204.14@tcp [ 1985.255597] LNet: Removed LNI 192.168.204.14@tcp2 [ 1985.304294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1985.394306] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1985.406673] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1985.410215] LNet: 116497:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1986.008875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1986.390229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1987.487656] LNet: Removed LNI 192.168.204.14@tcp [ 1987.506906] LNet: Removed LNI 192.168.204.14@tcp2 [ 1987.543453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1987.663444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1987.702721] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1988.470421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1988.487078] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1989.381845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1989.414895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1989.451124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1989.555048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 1989.926595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1991.008113] LNet: Removed LNI 192.168.204.14@tcp [ 1992.040270] LNet: Removed LNI 192.168.204.14@tcp2 [ 1992.089238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1992.190328] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 1992.205714] LNet: Added LNI 192.168.204.14@tcp2 [8/256/0/180] [ 1992.953687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 2000.705399] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2002.511613] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:22:38 (1763338958) [ 2002.602261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2002.656516] LNet: 117610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2002.660216] LNet: 117610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2002.666098] LNetError: 117610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2002.674464] LNetError: 117610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2002.678134] LNet: Removed LNI 192.168.204.14@tcp [ 2003.750489] LNet: Removed LNI 192.168.204.14@tcp2 [ 2003.803909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2004.267579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2004.290859] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 2005.388320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2005.443789] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2006.546129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2006.565500] LNet: Removed LNI 192.168.204.14@tcp [ 2006.578597] LNet: Removed LNI 10.1.2.3@tcp [ 2006.606254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2006.664844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2006.687933] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 2006.691798] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.213832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2007.240368] LNet: Removed LNI 192.168.204.14@tcp [ 2007.252640] LNet: Removed LNI 10.1.2.3@tcp [ 2014.858042] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:22:51 (1763338971) [ 2015.006605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2015.033951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2015.180366] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2015.383772] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2015.386145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2015.388798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2017.746840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2017.775503] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 2018.640361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2018.666744] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2021.111856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.14@tcp [ 2021.150604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2021.208866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.14@tcp -m PUT -e local_timeout -r 1 [ 2021.274805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2021.320928] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2021.323579] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2021.543145] LNetError: 118561:0:(lib-move.c:4947:lnet_parse()) 192.168.204.14@tcp, src 192.168.204.14@tcp: Dropping PUT (error -108 looking up sender) [ 2023.583257] LNet: Removed LNI 192.168.204.14@tcp [ 2023.597752] LNet: Removed LNI 10.1.2.3@tcp [ 2024.071576] Key type .llcrypt unregistered [ 2024.075109] Key type ._llcrypt unregistered [ 2034.412495] Key type ._llcrypt registered [ 2034.414324] Key type .llcrypt registered [ 2035.743940] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:23:12 (1763338992) [ 2036.047530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2036.072451] alg: No test for adler32 (adler32-zlib) [ 2036.973966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.358289] LNet: 119868:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.902472] Key type .llcrypt unregistered [ 2037.904259] Key type ._llcrypt unregistered [ 2043.112407] Key type ._llcrypt registered [ 2043.113737] Key type .llcrypt registered [ 2043.324789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2043.353768] alg: No test for adler32 (adler32-zlib) [ 2044.346522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2044.372901] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 2050.914438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 2051.097904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2051.564817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.114@tcp [ 2051.831095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2051.869519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2051.910460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2051.937792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2051.974511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2052.027466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2052.096541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2052.141620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2052.175650] LNet: 120314:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2052.209721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2052.267122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2052.318308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2052.347751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2052.384920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2052.427656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2053.192869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2053.415479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2053.443293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2053.471805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2053.496282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2053.520386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2053.555587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2053.586913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2053.600158] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.114@tcp [ 2053.627687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2053.644974] LNet: 120432:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2053.662834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2053.717545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2053.774725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2053.813955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2053.849417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2053.919733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2054.460263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.114@tcp [ 2054.640608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2054.671124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2054.706819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2054.739883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2054.760673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2054.794001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2054.817449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2054.843566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2054.858659] LNet: 120549:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2054.881816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2054.901910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2054.929962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2054.964942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2054.986869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2055.012252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2055.635528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2055.905502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2055.942217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2055.977872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2056.013965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2056.047638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2056.088354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2056.135722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2056.171040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2056.203941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2056.229557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2056.266268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2056.294971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2056.329788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2056.364679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2056.730787] LNet: 120711:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2056.739943] LNetError: 120711:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2057.784237] LNet: Removed LNI 192.168.204.14@tcp [ 2058.182354] Key type .llcrypt unregistered [ 2058.185237] Key type ._llcrypt unregistered [ 2071.783403] Key type ._llcrypt registered [ 2071.793254] Key type .llcrypt registered [ 2073.295311] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:23:49 (1763339029) [ 2073.776976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2073.787910] alg: No test for adler32 (adler32-zlib) [ 2074.769058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2075.141702] LNet: 121596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.772480] Key type .llcrypt unregistered [ 2075.776305] Key type ._llcrypt unregistered [ 2080.721672] Key type ._llcrypt registered [ 2080.723019] Key type .llcrypt registered [ 2080.928087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2080.940490] alg: No test for adler32 (adler32-zlib) [ 2081.873957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2081.909107] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 2087.464925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.114@tcp [ 2093.702253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.730837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2093.746731] LNet: Added LNI 192.168.204.14@tcp1 [8/256/0/180] [ 2094.683787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2095.370335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.114@tcp [ 2095.638959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2095.664337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2095.750698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2095.776837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2095.804321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2095.840641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2095.878740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2095.923727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2095.961439] LNet: 122244:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2095.979050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2096.035776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2096.085933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2096.128330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2096.183290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2096.232828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2097.133485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2097.314442] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.114@tcp, 192.168.204.114@tcp1 [ 2097.405977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2097.428736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2097.447376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2097.521520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2097.547160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2097.592097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2097.626094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2097.660936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2097.673445] LNet: 122361:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2097.690153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2097.724896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2097.759918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2097.808164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2097.861581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2097.894193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2098.545705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.114@tcp [ 2098.780927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2098.805892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2098.834001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2098.878936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2098.909584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2098.950957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2098.996919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2099.030505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2099.045734] LNet: 122478:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2099.072361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2099.100099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2099.156043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2099.196824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2099.247746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2099.299054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2099.985469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2100.277982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2100.302700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2100.339856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2100.376171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2100.423719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2100.481889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2100.540150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2100.585554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2100.628352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2100.661638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2100.697312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2100.727635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2100.768830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2100.815280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2104.478791] LNet: 122675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.482795] LNetError: 122675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.491115] LNet: Removed LNI 192.168.204.14@tcp [ 2104.499586] LNet: Removed LNI 192.168.204.14@tcp1 [ 2104.878382] Key type .llcrypt unregistered [ 2104.882380] Key type ._llcrypt unregistered [ 2117.325515] Key type ._llcrypt registered [ 2117.328211] Key type .llcrypt registered [ 2118.755299] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1893 sec ========== 19:24:35 (1763339075) [ 2119.265283] Key type .llcrypt unregistered [ 2119.268077] Key type ._llcrypt unregistered [ 2127.790600] Key type ._llcrypt registered [ 2127.791973] Key type .llcrypt registered [ 2128.138802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2128.149966] alg: No test for adler32 (adler32-zlib) [ 2129.164303] Lustre: Lustre: Build Version: 2.16.61_50_g1f7afa2 [ 2129.408069] LNet: Added LNI 192.168.204.14@tcp [8/256/0/180] [ 2131.079235] Key type lgssc registered [ 2131.894620] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2177.682373] Lustre: Mounted lustre-client [ 2182.380776] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2192.510849] LustreError: 125897:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff992020bbd000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2192.535153] LustreError: 125897:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2192.565823] Lustre: Unmounted lustre-client [ 2212.040184] Key type lgssc unregistered [ 2212.332417] LNet: 126380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.346084] LNetError: 126380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2212.372711] LNet: Removed LNI 192.168.204.14@tcp [ 2212.954163] Key type .llcrypt unregistered [ 2212.955734] Key type ._llcrypt unregistered